The string type is broken - <AUTHOR> http://mortoray.com/2013/11/27/the-string-type-is-broken/ <comment><USER2> The problem with text (that Unicode solves only partially) is that text representation, being a representation of human thought, in inherently ambiguous and imprecise.

Some examples:

(1) A == A but A != Α. The last letter is not uppercase "a", but uppercase "α". Most of the time, the difference is important, but sometimes humans want to ignore it (imagine you can't find an entry in a database since it contains Α that looks just like A). Google gives different autocomplete suggestions for A and Α. Is this outcome expected? is it desired?

(2) The Turkish alphabet is mostly the same as the Latin alphabet, except for the letter "i", which exists in two variants: dotless ı and dotted i (as in Latin). For the sake of consistency, this distinction is kept in the upper case as well: dotless I (as in Latin) and dotted İ. We can see that not even the uppercase <==> lowercase transformation is defined for text independently of language.

These are just two examples of problems with text processing that arise even before all the problems with Unicode (combining characters, ligatures, double- width characters, ...) and without considering all the conventions and exceptions that exist in richer (mostly Asian) alphabets.

<comment><USER3> I think (2) is an issue with Unicode specifically. They should have specified Turkish alphabet to use ı and a diacritic to make the dotted one. That would have made (in this case) capitalization locale-independent.

<comment><USER4> isn't this solving the wrong side of the problem? how about not having to think about such things at all and just accepting that uppercase/lowercase conversion is never going to be language agnostic.

thats futureproof and powerful, rather than extra thinking and work...

<comment><USER3> Most likely case-changes need to be locale-aware, that is true. But still I think minimizing number of locale-specifics is a reasonable goal and in that light I dislike the common usage of turkish i as a example because it is such a obviously fixable (if legacy stuff wasn't concern) flaw in Unicode rather than fundamental issue.

<comment><USER4> You are right, everything should be as easy as possible. This is a good philosophy for design in general...

</comment></comment></comment></comment></comment><comment><USER5> A nitpick from the article

 _> This spells trouble for languages using UTF-16 encodings (Java, C#, JavaScript)._

if they were using UTF-16, this wouldn't be a problem as UTF-16 can be used to perfectly well encode code points outside of the BMP (at the cost of losing ability for O(1) access to specific code points of course. If you need to know what the n-th code point is, you have to scan the string until the n-th position).

They are, however, using UCS-2 which can't. If you use a library that knows about UCS-2 to work on strings encoded in UTF-16, then you will get broken characters, your counts will be off and case transformations might fail.

Most languages that claim Unicode support still only have UCS-2 libraries (Python 3 is a notable exception)

<comment><USER6> > Most languages that claim Unicode support still only have UCS-2 libraries > (Python 3 is a notable exception)

Most non-JVM languages[1] actually use UTF-8 as the internal encoding so they should not suffer from this. Python 3 does not use UTF-16 either, it selects an encoding based on the contents of the string.

[http://www.python.org/dev/peps/pep-0393/](http://www.python.org/dev/peps/pep-0393/)

1\. I think .NET too uses UCS-2 or UTF-16, but I am not a Windows developer.

<comment><USER7> Python <3.3 uses UCS2 or UCS4, depending on the build

Ruby >1.8 lets you choose the encoding

.NET UCS2/UTF-16 (I know the difference, imho if the stdlib has a .size, .length or .count that works on code units instead of code points it's broken... thus I'll mention only UCS2 from now on)

Java UCS2

Clojure UCS2

Scala UCS2

QT UCS2

Haskell String UCS4

Haskell Data.Text UTF-16 (yes, not a naive UCS-2)

Rust UCS4 (last time I checked)

Javascript UCS2

Dart UCS2

PHP Unicode-oblivious

Vala UCS4

Go UTF-8 (but it lets you call len() on strings, and it doesn't return the length of the string, but its size in bytes)

I can't really think of another language that uses UTF-8 internally, are you sure?

<comment><USER8> > Rust UCS4 (last time I checked)

Rust chars are 32bit Unicode codepoints. But strings themselves are utf-8. That is the string type, ~str, is basically just ~[u8], a vector of bytes and not ~[char].

`.len()` [O(1)] gives you byte length while `.char_len()` [O(n)] gives you the number of codepoints.

So strings in rust are just vectors of bytes with the invariant that it's valid utf-8.

<comment><USER7> Thanks, I didn't know that

</comment></comment></comment></comment></comment><comment><USER5> Python 3 gets so much of this right. It's one of the things I really loved about python 3 as it allows for correct string handling in most cases (see below).

Note that this is only really true with Python 3.3 and later as in earlier versions stuff would start breaking for characters outside of the BMP (which is where JS is still stuck at, btw) unless you had a wide build which was using a lot of memory for strings (4 bytes per character)

In general, internally using unicode and converting to and from bytes when doing i/o is the right way to go.

But: Due to [http://en.wikipedia.org/wiki/Han_unification](http://en.wikipedia.org/wiki/Han_unification) being locked into Unicode with a language might not be feasible for everybody - especially in Asian regions, Unicode isn't yet as widely spread and you still need to deal with regional encodings, mainly because even with the huge character set of Unicode, we still can't reliably write in every language.

Ruby 1.9 and later helps here by having many, many string types (as many as it knows encodings), which can't be assigned to each other without conversion.

This allows you to still have an internal character set for your application and doing encoding/decoding at i/o time, but you're not stuck with unicode if that's not feasible for your use-case.

People hate this though because it seems to interfere with their otherwise perfectly fine workflow ("why can't I assign this "string" I got from a user to this string variable here??"), but it's actually preventing data corruption (once strings of multiple encodings are mixed up, it's often impossible to un- mix them, if they have the same characer width).

I don't know how good the library support for the various Unicode encodings is in Ruby though. According to the article, there still is trouble with correctly doing case transformations and reversing them.

Which brings me to another point: Some of the stuff you do with strings isn't just dependent on string encoding, but also locale.

Uppercasing rules for example depend on locale, so you need to keep that into account too. And, of course, deal with cases when you don't know the locale the string was in (encoding is hard enough and most of the cases undetectable - but locales - next to impossible).

I laugh at people who constantly tell me that this isn't hard and that "it's just strings".

<comment><USER9> > _Python 3 gets so much of this right_

What does it gets right????? It's all broken as nearly everything else!

It's sad 99% comments there are "oh see, I can run _some_ examples from page just fine. So everything's all right, I've got full Unicode!"

The reality is there's 1-2 languages that are trying to make it correct from the beginning (perl6, I'm looking at you). It's 2013 and if language can compose bytes to code points everyone declares a win, sticks "full unicode support" label to it and continues to _str[5:9]_.

"But I've got UnicodeUtils!" — it won't help. People just don't want or cannot write it correctly. Word is not [a-z]. Not [[:alpha:]] either. And not [insert regex here]. You cannot reverse by reversing codepoint list. And you cannot reverse by reversing grapheme list. And string length is hard to compute and then it doesn't any sense. And indexing string doesn't make any sense and it's far away from O(1)

<comment><USER10> Can you provide some examples of Python 3 getting strings wrong?

Between strings being native unicode code points (you have to encode to bytes to get UTF-8) and unicodedata for normalization and decomposition ([http://docs.python.org/3.3/library/unicodedata.html](http://docs.python.org/3.3/library/unicodedata.html)) I've found Python 3 pretty robust. Python 3.3 also uses appropriate Unicode data for regular expressions, as mentioned on [http://docs.python.org/3.3/howto/regex.html](http://docs.python.org/3.3/howto/regex.html).

If you want to compare strings you should really normalize them first, which is where unicodedata comes in. In my programming situations it would be wrong to conflate different decomposition of the same unicode string. Why is this? Because other software you interact with uses encodings and the UTF-8 encoding of two different decompositions if different. I've run into this with UTF-8 filenames on OS X when working with Subversion.

<comment><USER9> Did you read the comment you're replying to at all? You can start at "It's sad 99% comments".

PS:

      Python 3.3.2 (default, Nov 27 2013, 20:04:48)
      [GCC 4.2.1 Compatible Apple LLVM 5.0 (clang-500.2.79)] on darwin
      Type "help", "copyright", "credits" or "license" for more information.
      >>> 'öo̧'[1:]
      '̈o̧'

And sorry, those new regexes don't even support \X (grapheme matching)

Edit: python version

<comment><USER10> Yes, I did, and you did not provide a single example. You just said ""oh see, I can run some examples from page just fine. So everything's all right, I've got full Unicode!".

Taking the time to actually prove your point it useful. However, your recent example seems to be running fine on Python 3.3. You did not include any version info in your example output.

        Python 3.3.0 (default, Mar 11 2013, 00:32:12)
        [GCC 4.7.2] on linux
        Type "help", "copyright", "credits" or "license" for more information.
        >>> "öo̧"[1:]
        'o̧'
        >>>

I haven't run across any situations where Python 3.3 is doing wrong, which is why I am asking for some examples.

<comment><USER9> 3.3.2. No, it is not. Use 'o\u0308o\u0327'

<comment><USER10> Oh, I see the issue here. You are expecting the string class to function via graphemes rather than characters. It should be possible to implement grapheme support since character support is there, but I imagine the reverse is not true.

A little googling turned this up. [https://mail.python.org/pipermail/python- ideas/2013-July/021...](https://mail.python.org/pipermail/python- ideas/2013-July/021916.html)

<comment><USER11> TL;DR of parent comment here for those skimming:

      x = 'o\u0308o\u0327'
      len(x) == 4
      x == "öo̧"
      x[:2] == "ö"
      x[2:] == "o̧"
      x[0] == x[2] == "o"

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER12> The string type isn't broken. If anything these "X is broken" posts are broken. Taking one special case, finding problems with that case and deducing that the whole concept must therefore be discarded is just silly. Strings work fine for the vast majority of use cases. No technology is free of flaws and engineering decisions are almost always based on weighting the pros and cons and choosing a solution that on balance works best. Strings are a useful feature and Unicode is a notoriously hard problem. Proposing to go back to arrays of characters makes things worse for most people in most cases and therefore is not a practical solution.

<comment><USER13> In my experience, the world is full of software which "work fine for the majority of use cases" until the point where you take the wrong code path and things go south.

<comment><USER14> Much like human brains, and business processes.

</comment></comment></comment><comment><USER15> In many languages it's difficult fixing the string type without breaking existing code. In Ruby: String#upcase only handles ASCII (by spec), #length counts codepoints, #reverse reverses codepoints.

You can use UnicodeUtils if you need "full" Unicode support:

        >> UnicodeUtils.upcase("baffle")
        => "BAFFLE"
        >> graphemes = UnicodeUtils.each_grapheme("noe\u0308l").to_a
        >> graphemes.reverse.join
        => "lëon"
        >> graphemes.size
        => 4
        >> graphemes[0, 3]
        => "noë"

<comment><USER9> > _String#upcase only handles ASCII (by spec)_

Bad for Ruby

> _You can use UnicodeUtils if you need "full" Unicode support:_

Oh, sure

      Betty:~ lelf$ ruby -r unicode_utils/u -e 'puts UnicodeUtils.each_grapheme("A͜CBD").to_a.reverse.join'
      DBC͜A

So, "full" (it's not) Unicode support won't help you if you have little idea about what you're doing (like indexing stringه҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿s)

<comment><USER16> what are these characters printing here?

<comment><USER17> It's an awesome little gadget - looks like one character, but is a really big messy bunch of bytes:

"\xD9\x87\xD2\x88\xCC\xBF\xD2\x88\xCC\xBF\xD2\x88\xCC\xBF\xD2\ x88\xCC\xBF\xD2\x88\xCC\xBF\xD2\x88\xCC\xBF\xD2\x88\xCC\xBF\xD 2\x88\xCC\xBF\xD2\x88\xCC\xBF\xD2\x88\xCC\xBF\xD2\x88\xCC\xBF\ xD2\x88\xCC\xBF\xD2\x88\xCC\xBF\xD2\x88\xCC\xBF\xD2\x88\xCC\xB F\xD2\x88\xCC\xBF\xD2\x88\xCC\xBF\xD2\x88\xCC\xBF\xD2\x88\xCC\ xBF\xD2\x88\xCC\xBF\xD2\x88\xCC\xBF\xD2\x88\xCC\xBF\xD2\x88\xC C\xBF\xD2\x88\xCC\xBF\xD2\x88\xCC\xBF\xD2\x88\xCC\xBF\xD2\x88\ xCC\xBF\xD2\x88\xCC\xBF\xD2\x88\xCC\xBF\xD2\x88\xCC\xBF\xD2\x8 8\xCC\xBF\xD2\x88\xCC\xBF\xD2\x88\xCC\xBF\xD2\x88\xCC\xBF\xD2\ x88\xCC\xBF\xD2\x88\xCC\xBF\xD2\x88\xCC\xBF\xD2\x88\xCC\xBF\xD 2\x88\xCC\xBF\xD2\x88\xCC\xBF\xD2\x88\xCC\xBF\xD2\x88\xCC\xBF\ xD2\x88\xCC\xBF\xD2\x88\xCC\xBF\xD2\x88\xCC\xBF\xD2\x88\xCC\xB F\xD2\x88\xCC\xBF\xD2\x88\xCC\xBF\xD2\x88\xCC\xBF\xD2\x88\xCC\ xBF\xD2\x88\xCC\xBF\xD2\x88\xCC\xBF\xD2\x88\xCC\xBF\xD2\x88\xC C\xBF\n"

<comment><USER16> funny that an ad comes up when you search that character on google. [http://imgur.com/Y5vkbHF](http://imgur.com/Y5vkbHF)

</comment></comment></comment></comment></comment><comment><USER18> Perl seems to pass nearly all the tests (including uppercasing baffle):

      $ perl -E 'use utf8; binmode STDOUT, ":utf8"; say uc("baffle");'

BAFFLE

The only failure I can see is that it treats "no<combining diaresis>el" as 5 characters (so reports length as 5 and reversing places the accent on the wrong character). That's documented here: [http://perldoc.perl.org/perluniintro.html#Handling- Unicode](http://perldoc.perl.org/perluniintro.html#Handling-Unicode) "Note that Perl considers grapheme clusters to be separate characters"

All else seems to work though (including precomposed/decomoposed string equiality etc). The docco also says that perl's regex engine with Do The Right Thing with matching the entire grapheme cluster as a single char.

<comment><USER19> Python 3 gets that one, but python 2.7 doesn't:

        $ python3 -c 'print("baffle".upper())'
        BAFFLE
        $ python -c 'print "baffle".upper()'
        BAfflE
        $ python -c 'print u"baffle".upper()'
        BAϬ"E

<comment><USER20> It's interesting that you get BAFFLE for the first one. I get the same result in both 3 and 2.

Note first that the reason you get "BAϬ"E" is a bit of garbarge-in garbage- out. Strangely, the interpreter isn't rejecting that with the typical "SyntaxError: Non-ASCII character <char> in file" error; instead, it appears to be assuming ISO-8859-1, and then performing .upper(). You can fix that:

        python2 -c '# coding: utf-8
        print u"baffle".upper()'

(Note, of course, that the #coding needs to match your terminals encoding, which is likely UTF-8, but it isn't guaranteed.)

That, for me, prints "BAfflE" in both Python 2 and 3 (adjusting for 3 by adding parens around print, and removing the u prefix on the literal.) I'm on Python 3.2, so perhaps 3.3 does better. (I'm behind on updates, but last I did update, Gentoo stable was still on 3.2.)

<comment><USER19> Interesting.

I am on python 3.3, but I don't know if it's the updated interpreter that fixes the bug.

</comment></comment></comment></comment><comment><USER21> ˙ƃuᴉuɐǝɯ ⅋ 'spɹoʍ 'sɥdʎlƃ 'sɹǝʇɔɐɹɐɥɔ uǝǝʍʇǝq 'ɹǝʌǝʍoɥ 'ǝɔuǝɹǝɟɟᴉp ɐ sᴉ ǝɹǝɥ┴ ˙ʇxǝʇ ɥʇᴉʍ punoɹɐ ƃuᴉsɹɐ oʇ sǝɯoɔ ʇᴉ uǝɥʍ sǝᴉʇᴉlᴉqᴉssod ƃuᴉʇsǝɹǝʇuᴉ ǝɯos sɹǝɟɟo ǝpoɔᴉu∩

<comment><USER22> Awesome way to exercise the brain.

<comment><USER23> Interesting. I had no problem reading that (except for 'arsing' which I thought I'd misread). The ability of the brain to pattern-match upside down is amazing.

<comment><USER24> Fascinatingly, I read your comment first, then tried to read the upside down post — the only word I had trouble with was arsing.

Sight reading is really fascinating.

</comment></comment></comment></comment><comment><USER25> I think the mistake here is seeing a string as an extension of an array or vector. What I would prefer is a string type that didn't support all the operations of vectors. The length of a string is not inherently a meaningful question (and for the cases where it is, what you want is something like a vector of grapheme clusters - which is a useful type to have, but not so useful that every string in your program should incur the overhead of creating such a thing); likewise reversing and splitting are operations that simply shouldn't be allowed for your "fast path, undecoded string" type.

<comment><USER26> I'm with you here; but in that case, I'd like an ascii_string type, which most languages don't provide specifically. This type _would_ support string reversal, substring slices, and so on, but be limited to 7-bit ASCII only. I think there are many use cases that are purely internal, and don't need i8n. It's handy to be able to do things, including operations on strings, for internal things. Filename handling where you control the filename, the "keys" in languages which use strings for a dictionary type, and so forth.

<comment><USER6> I think this might just confuse new programmers and the filename thing is especially dangerous since at some point you might want to support i18n there. I think it would be better to have two types of string: 1) unicode strings and 2) arrays of 8 byte data with some string like functions (essentially C strings). The second case is essentially binary data strings.

</comment></comment></comment><comment><USER27> A big problem here is a lack of clear definitions for various concepts like "character," "reversed string," "upper case," etc. The author briefly recognizes this, but brushes it off with statements like "I generally expect that..." and "I assume most people would not be happy with the current result."

I think these hand-wavings aren't helpful. Short of extensive surveying, which is bound to be controversial no matter what the result, talking about "general expectations" is a purely subjective notion, and not a good way to evaluate the actions of cold, soulless silicon that is just following orders.

Like the author, I also consider myself a mostly reasonable person, yet is might come up with very different expectations. If I saw that "ffl" ligature, how would I know it's a ligature and not some single unrelated character in another language? You might respond "but it's clearly part of the word 'baffle' and should be capitalized thusly." But would you suggest that string libraries ship with word lists and perform contextual analysis to determine how to perform string operations? Surely that's a fool's errand, not to mention that it would inevitably produce _unexpected_ results.

<comment><USER28> "If I saw that "ffl" ligature, how would I know it's a ligature and not some single unrelated character in another language?"

Because the name of that character is "Latin Small Ligature ffl". Knowing to capitalize ffl as FFL doesn't require a word list any more than knowing to capitalize "ffl" does.

</comment></comment><comment><USER29> I'm not sure I agree with the title, although I do agree with just about all of the content:

* a string type is probably a good idea to bundle the subtleties of unicode, a plain array or list (whether it's of bytes or of codepoints) won't cut it: standard array operations are incorrect/invalid on unicode streams

* the _vast_ majority of string types are broken anyway, as even in the best case they're codepoint arrays (possibly with a smart implementation). The bad cases are just code unit arrays, which break before you even reach fine points of unicode manipulation

And then, you've got the issue that a lot of unicode manipulation is locale- dependent, which most languages either ignore completely or fuck up (or half and half, for extra fun)

</comment><comment> b-johansson If you are actually manipulating strings rather than just storing and pushing them around I would suggest looking at ICU. Handling Unicode is difficult and it's easy to confuse encodings, code points and glyphs or make assumptions based on your own culture and language.

ICU has support for a lot of the basic operations you would want to perform on strings as well as conversion to whatever format is suitable for your platform and environment.

</comment><comment><USER30> Do people really need to reverse strings in the real world?

I don't think I've ever written code to do that outside of homework assignments and interviews.

<comment><USER31> May be not reversing, but trimming a Unicode string to certain character count is a close relative and it is a very common operation.

<comment><USER6> What do you use it for? Unless you have a monospaced font the number of characters do not mean much. So unless you are implementing command line tools or text editors it should not be that common.

<comment><USER32> I have a database field limited to 100 "characters" [1]. The user sent me a form submission with 150. I need to do something to resolve that. This is _incredibly_ common. Truncation to a defined size is routine.

[1]: I'm leaving "characters" undefined here, because no matter what Unicode- aware definition you apply here, you've got trouble.

<comment><USER33> "I have a database field limited to 100 "characters"."

Well there's your problem right there...

"The user sent me a form submission with 150. I need to do something to resolve that."

Any software that defines "do something" here as "silently discard 1/3rd of the user input" is software I'm going to throw in the trash. If you must have fixed-length fields, surely telling the user "much characters, wow overflow" is better than just chopping the input.

<comment><USER32> Since this seems to be confusing people, I'm providing a small hypothetical example here.

"Any software that defines "do something" here as "silently discard 1/3rd of the user input" is software I'm going to throw in the trash."

You are reading far more in than I put in. I merely said _somehow_ you need to resolve this; you put a particular resolution in my mouth, then attacked.

I did choose the web for one reason, which is that you can't avoid this case; you can try to limit the UI to generating 100 characters only (and I still haven't defined "characters"...), but it's 15 seconds for a user to pull open Firebug and smash 150 characters into your form submission anyhow. _Somehow_ , you better resolve this, and as quickly as you mounted the high horse when faced with the prospect of mere truncation, throwing the _entire_ request out for that will cause somebody else to mount an equally high horse....

</comment></comment></comment></comment></comment></comment><comment><USER34> This is why the U.S. dominates the software world. Back when everyone was figuring out how to express their languages, we had the option to punt on complexity and just use ASCII.

<comment><USER35> ASCII doesn't make the U.S. special. ASCII is special because it's from the U.S.

Lots of people speak languages that trivially fit in 8 bits with no real "figuring out" to do. Before Unicode, we all had our different codepages or encodings. Including the U.S.

The U.S. is pretty central to computing. Because of that, and because ASCII only uses 7 bits, some other 8-bit cultures use it as a subset for their native 8-bit encodings. Even in the U.S, we use extensions to ASCII so we can represent text in languages that are close cousins to English. I doubt you actually use ASCII much. You've probably been using either ISO 8859-1 (aka Latin-1), which is a superset of ASCII, or Windows-1252, which is a superset of Latin-1.

[http://msdn.microsoft.com/en- us/library/cc194884.aspx](http://msdn.microsoft.com/en- us/library/cc194884.aspx)

This mess of incompatible codepages and culture specific encodings is one of the main problems that Unicode was invented to solve. It also happens to help languages which need more than 8 bits.

<comment><USER34> Many languages fit into 8 bits, but English is particularly simple in its alphabet. Even many of the European languages that can fit in 8 bits have things like accented characters that complicates things somewhat.

Of course this isn't to say English is simple overall. Just that it's complexities lie elsewhere, and it's simplicities lie in an area that made it particularly simple for early computer systems to process.

<comment><USER36> > Even many of the European languages that can fit in 8 bits have things like > accented characters that complicates things somewhat.

I don't see your point here, with respect to English orthography making computer implementation easier. How exactly does not needing representations for accented characters make anything easier?

<comment><USER37> If it was just some additional characters like ñ (which is considered a letter of its own, not an accented n) then it wouldn't be a big deal – but e and é are the same letter with different accents, which adds some subtlety that English simply doesn't have. Given a small enough number of accented characters you can punt on that, call them each a character, but English is objectively simpler since the only real distinction it has between letters is caps or not-caps. (I was just watching the Mother Of All Demos, though, and everything was in caps but they put an overline over capital letters. So even normal English lettering was too complicated for a while.)

</comment></comment></comment></comment></comment><comment><USER38> Hat tip to Guido van Rossum for passing (nearly) all the tests in Python 3.

Is the "ffl-ligature to uppercase" test really relevant? Isn't that fixed by appropriate use of string normalisation?

<comment><USER39> I also doubt the validity of the upper-casing, it feels like in an internationalization/localization context, converting a string to all upper case is not a valid thing to be doing.

Not all languages (or even characters) _have_ a well-defined upper-case versions of their glyphs.

Even if they all did, I would expect the interpretation by a (human) reader to vary culturally.

<comment><USER40> The usual goal is to apply a consistent transform though, to smooth out interpretation differences - i.e. when looking for command input I either lowercase or uppercase things to smooth over the fact that "yes" "YES" "Yes" are all completely valid ways of saying the same thing with those characters.

If there's only one way of expressing the thing - i.e. a single chinese character - then it would be valid to do nothing. It's just in english "y" and "Y" might change context, but as far as computer input is generally concerned they are the same thing.

<comment><USER41> For that use case it is better to compare case insensitively with "yes" instead of converting the input to lower case first.

<comment><USER3> How do you do case-insensitive comparison without normalizing the case of the operands?

<comment><USER42> Most string compare routines in the library offer a case insensitive compare option already, you don't have to normalize it.

</comment></comment></comment></comment></comment></comment><comment><USER43> While JavaScript (in browsers) has no way to normalize precomposed/decomposed strings, it has standard methods to correctly compare them: [https://developer.mozilla.org/en- US/docs/Web/JavaScript/Refe...](https://developer.mozilla.org/en- US/docs/Web/JavaScript/Reference/Global_Objects/String/localeCompare) [https://developer.mozilla.org/en- US/docs/Web/JavaScript/Refe...](https://developer.mozilla.org/en- US/docs/Web/JavaScript/Reference/Global_Objects/Collator)

E.g.:

        var decomp="noël";
        var precomp="noël";
        console.log(decomp.split(""));
        console.log(precomp.split(""));
        console.log(decomp.localeCompare(precomp));

Prints:

        ["n", "o", "e", "̈", "l"]
        ["n", "o", "ë", "l"]
        0

Browser support for this varies, The Intl.Collator interface is currently only supported Chrome (maybe also in Opera? Idk).

Note: In Chrome when comparing (e.g. sorting) a lot of strings String.prototype.localeCompare is much slower than using a pre-composed Intl.Collator instance (because internally localeCompare creates a new collator for each call). Using Intl.Collator rediced startup time of my [http://greattuneplayer.jit.su/](http://greattuneplayer.jit.su/) immensely. node.js currently has no support for Intl.*. It probably will be a compile time option for 0.12.

</comment><comment><USER44> This article is mostly written from a European language perspective. For Indian scripts, storing combining characters as a separate code points is the right thing to do.

For example, कि (ki) is composed of क and ि When I'm writing this in an editor, say, I typed ku (कु) instead of ki (कि) and I press backspace, I indeed want to see क rather than deleting the whole "कि".

<comment><USER45> Only some times I figure, because if you want to make the first letter green, you'd want that to apply to the whole कि.

</comment></comment><comment><USER46> For the record, Racket gets the "baffle" example right:

        racket@> (string-upcase "baffle")
        "BAFFLE"

It also passes all of the author's other tests (except for the ones involving combining diacritics, but racket includes built-in functions for normalizing such strings so you can work with them)

</comment><comment><USER47> I seem to have rather little use for the cases the author presents here. If I'm working with strings, they are either of the debug or internal variant, where even basic ASCII would suffice, or I get them from somewhere and don't touch them at all, just pass them around.

But what I absolutely need in a language is to have a very very clear seperation between _strings_ and _byte arrays_ , or raw data, and ideally a way to transform between the two. C# gets this right with its byte and string types, the framework uses them correctly, and there is the wonderful Encoding namespace to interchange the two. Python 2.7 is the absolte worst, it's apparently impossible to get anything done with raw data and _not_ run into some obscure 'ASCII codec can't handle octet 128' whatever exception (reminds you why we have strict typing: magic is fucking annoying).

</comment><comment><USER48> I'd have hoped Common Lisp would fare well here, but SBCL (1.1.11 on 64-bit Linux Mint 15) is pretty broken. My results:

string: noël, reversed: l̈eon, first 3 chars: noe, length: 5

string: 😸😾, reversed: 😾😸, first 1 char: 😸, length: 2

string: baffle, upcase: BAfflE

string: noël, equals precomposed: NIL

 _Edited_ : GNU CLISP 2.49 produces identical results.

<comment><USER49> I was somewhat disappointed as well. I wrote some tests here: [http://paste.lisp.org/display/140280](http://paste.lisp.org/display/140280)

Perhaps playing around with different internal representations as pointed out by sedachv ([https://news.ycombinator.com/item?id=6811407](https://news.ycombinator.com/item?id=6811407)) would work but the initial, naive string usage doesn't work.

While I expected the default usage to work correctly in Common Lisp.

</comment></comment><comment><USER50> What we really should be doing is doing away with broken nomenclature.

What does the "length" of a string even mean? A database will tell you it has to do with storage. A nontechnical person will say it's the number of symbols. A visual designer might say that it has to do with onscreen width when rasterized in a particular way. None of these people are obviously right or wrong.

It's very useful to be able to count the number of glyphs in a string, or the number of unicode codepoints, or bytes, or pixels when rasterized in a particular way, but "length" isn't clear enough to unambiguously refer to any of them. Any meaning you try to ascribe to the "length" operation is going to be wrong to someone.

</comment><comment><USER51> All of these examples work in Haskell's canonical text library, 'text'! It's the only language I know of that works.

<comment><USER52> The reversal of the decomposed noël doesn't produce the right result. Converting baffle to uppercase does do the right thing though, and the rest works as expected.

</comment></comment><comment><USER53> I don't think the solution to this problem is to make our string classes more complicated. I think it's to make our languages and character sets less complicated. I can't believe that multiple codepoints being used to generate a single glyph made it into the Unicode spec. That breaks a bunch of extremely useful abstractions. I think it is reasonable to expect human languages to be made up of distinct glypths that do not interfere with each other. Any language that does not is too complicated to be worth supporting. Let it die.

</comment><comment><USER54> Now let's take the lower case of "BAFFLE" \- should we get "baffle" or should the string class/function/wtfe attempt to recognize that a ligature can replace "ffl" and return to us "baffle"? More generally, should the string library ever attempt to replace letter with ligatures? Should this be yet another option?

And as I type this, another issue manifests: the spelling correction can't even recognize baffle as a properly spelled word; it highlights the 'ba' and ignores the rest.

<comment><USER55> Uppercasing and lowercasing is inherently lossy. E.g. the German ß becomes SS when uppercased, yet there is no way to know whether SS should be lowercased to ss or ß again. That's a reason why those things should be used, if at all, only as display transformations. Same goes for ligatures, but even those actually shouldn't be applied automatically, depending on the language. E.g. in German ligatures cannot span syllables and few layout engines can detect that.

<comment><USER3> I feel like I should learn German only so that I would be able to comment on the ß issue every time a Unicode thread pops up. From my uninformed point of view it is not really clear if ß should really be handled as a separate character/grapheme, or just as a ligature in rendering phase and stored as 'ss'. Or even if current-day orthography should be held at such a sacrosanct position that it shouldn't be changed to save significant amount of collective effort.

<comment><USER56> > or just as a ligature in rendering phase and stored as 'ss'.

Probably.

> to save significant amount of collective effort

I've seen this kind of suggestion a number of times on HN, and I find it highly amusing. When confronted with a difficult challenge in representing the world on a computer, apparently the answer is to instead change the world.

OK, but then how are you going to handle hundreds of years of legacy texts?

<comment><USER57> In German, 'ß' is definitely not just a ligature of 'ss'.

Consider 'Masse' (mass) vs. 'Maße' (dimensions).

Uppercasing these words will necessarily produce ambiguity.

It would be equally tempting -- and wrong -- to treat the German characters 'ä', 'ö' and 'ü' as ligatures of 'ae', 'oe' and 'ue'. They're pronounced the same, and the latter forms commonly occur as substitutions in informal writing, but they also occur in proper names, where it would be incorrect to substitute them with the former. However, if you want to sort German strings, 'ä', 'ö' and 'ü' sort as 'ae', 'oe' and 'ue'.

</comment></comment></comment></comment></comment><comment><USER33> If anybody hasn't seen it, Glitchr's twitter is a fantastic example of how bizarro things can get with "140 characters".

[https://twitter.com/glitchr_](https://twitter.com/glitchr_)

Note: may freak out browsers with a flaky Unicode implementation. For instance, scrolling that stream on the iOS Twitter client can get very laggy.

</comment><comment><USER58> For Go: the for-range loop iterates 5 times, reversed (manually, using the resulting runes) is l̈eon, utf8.RuneCount is 5. The blog has just recently been talking about text normalization[1] via a library, but it isn't built into the core.

[1] [http://blog.golang.org/normalization](http://blog.golang.org/normalization)

</comment><comment><USER44> The author intentionally chooses decomposed form. Indeed all of them work with Python 3. Here:

        Python 3.3.2+ (default, Oct  9 2013, 14:50:09)
        [GCC 4.8.1] on linux
        Type "help", "copyright", "credits" or "license" for more information.
        >>> noel="noël"
        >>> noel[::-1]       # reverse
        'lëon'
        >>> noel[0:3]        # first three characters
        'noë'
        >>> len(noel)        # length
        4

The point is, defining what is a character based on how it is displayed is flawed. Just precompose the string ifg you want and carry on. Like I said in my other comment, making automatic conversion of decomposed -> precomposed wrecks havoc with Indian languages.

<comment><USER59> Works as expected too in Scala, although it might be because the terminal does normalization.

scala> val noel = "Noël" noel: String = Noël

scala> noel.reverse res0: String = lëoN

scala> noel.take(3) res1: String = Noë

scala> noel.length res2: Int = 4

scala> import java.text.Normalizer

val nfdNoel = Normalizer.normalize(noel, Normalizer.Form.NFD) import java.text.Normalizer

scala> nfdNoel: String = Noël

scala> nfdNoel.length res3: Int = 5

scala> nfdNoel.reverse res4: String = l̈eoN

scala> nfdNoel.take(3) res5: String = Noe

The problem with an array of characters, as he mentions, is that it doesn't work properly in many use cases. If your array of characters stores 16 bit codepoints, it breaks with the 32 bit codepoints (Java got bit hard by that, where a char used to be a character prior to the introduction of surrogate pairs in Unicode); if it stores 32 bit codepoints, then it's pretty wasteful in most cases, which is exactly why you'd want a string type that handles storage of series of characters in an optimal fashion.

</comment></comment><comment><USER60> I hope Haskell Prime solves this. In Haskell String is literally a list of characters. This causes some overhead and leads to bad performance. Of course we got Text and for binary data you can use ByteString, but it's a bit of pain compared to having a real string type by default.

</comment><comment><USER61> I think the specific case of ligatures isn't a failure in strings per se, but a failure in Unicode in that it includes them in the first place. What "fi".upper() (or whatever) should do is kind of ambiguous. The following doesn't really seem appropriate:

      "fi".upper().lower() #=> "fi"

But obviously nor does

       "fi".upper() #=> "fi"

In Turkish (which distinguishes between dotted and dotless 'i'), this issue exists already:

       "ı".upper().lower() #=> "i"

This case _couldn 't_ (so far as I know) be fixed by any string library without breaking Unicode compatibility, so it seems slightly disingenuous to call it an issue with strings.

</comment><comment><USER62> Tom Christiansen (of Perl fame) made a much, much thorough analysis of Unicode problems in his OSCON 2011 presentation: [http://www.oscon.com/oscon2012/public/schedule/detail/24252](http://www.oscon.com/oscon2012/public/schedule/detail/24252)

Here are the slides: [http://training.perl.com/OSCON2011/gbu/gbu.pdf](http://training.perl.com/OSCON2011/gbu/gbu.pdf)

The site seems down ATM, but Internet Archive has it: [https://web.archive.org/web/20121224081332/http://98.245.80....](https://web.archive.org/web/20121224081332/http://220.127.116.11/tcpc/OSCON2011/gbu.html)

</comment><comment><USER63> The article briefly mentions JavaScript, which uses something similar to UTF-16/UCS-2: [http://mathiasbynens.be/notes/javascript- encoding>](http://mathiasbynens.be/notes/javascript-encoding>)

Here's a slightly more in-depth blog post on the many issues this causes, and how to avoid them in JavaScript: [http://mathiasbynens.be/notes/javascript- unicode](http://mathiasbynens.be/notes/javascript-unicode) Some of these problems are briefly mentioned in the above post, too.

</comment><comment><USER64> This is misinformation. OP's strings are just wrong...

        >> "\u0308"
        => "̈"
        >> "\u00eb"
        => "ë"
        >> "noe\u0308l"
        => "noël"
        >> "no\u00ebl"
        => "noël"

His noël examples work just fine if you don't copy/paste the string he posts, and instead type them in like I just did.

If anything, languages are reporting correct reverses and length, since he's really manipulating 5 characters rather than four.

<comment><USER65> Congratulations, you've discovered unicode composition!

      2.0.0p247 :045 > Unicode::compose("e\u0308").unpack('U').first.to_s(16)
       => "eb"
      2.0.0p247 :046 > Unicode::compose("\u00eb").unpack('U').first.to_s(16)
       => "eb"
      2.0.0p247 :047 > Unicode::decompose("e\u0308").unpack('U').first.to_s(16)
       => "65"
      2.0.0p247 :048 > Unicode::decompose("\u00eb").unpack('U').first.to_s(16)
       => "65"

I presume the ones you pasted in were changed by the browser. His examples are not wrong at all, indeed how can a string be "wrong"?

<comment><USER64> His example is "wrong" in the sense that you cannot reasonably complain that "noe¨l" gets reversed to "l¨eon" and put the "¨" part on top of the "l" when it does — which seems entirely correct. Or for that matter, that the string's length is 5 when there are indeed 5 characters.

As for being changed by the browser, the latter (or rather the OS) copied what there was, and the OS pasted it verbatim insofar as I can tell.

</comment></comment></comment><comment><USER66> Honestly I think 'you' computer programmers love useless challenges too much. Why can't you adopt lessons from Q?

If it isn't easy to get some languages working with Unicode properly then fix the languages and leave Unicode alone. Remove all the language characteristics that makes working with Unicode difficult. If Unicode will not go to the language then the language must go to Unicode, or opt out of the computer era, or die!!

KISS!!

</comment><comment><USER67> There is a one more issue -- the easier it is to manipulate strings in some language the greater chance that they will be used as an internal data structure for things that certainly aren't texts. And this almost always causes substantial performance loss and awful bugs that are either untraceable due to a dependence on subtle configuration details or form security holes. Or both.

</comment><comment><USER68> I think a lot of programmers don't properly understand character encoding simply because their programming languages don't give them the proper treatment. We need more APIs that force developers to acknowledge character encodings, probably in the type system.

</comment><comment><USER4> this hits on one of my biggest problems with native android and ios development. the wcs/wchar functions are largely broken or unusable... it caused me a real headache from not knowing upfront.

the idea of the string type is just fine though (or a character array) broken implementations don't invalidate it, they just invalidate the myth of '3rd party libraries must be good because hundreds of programmers worked on them for years' \- which is exactly a myth. it doesn't just apply to strings but everything. (not brokeness, just that you shouldn't expect them to work beyond what you can measure, and certainly shouldn't expect that they are flawless or even good implementations)

</comment><comment><USER69> Out of curiosity, why only have one string type? We don't do the same for numbers. Many languages don't have "number", they have int, float, long, etc.

Instead of just String, maybe we should have ASCIIString, UTF8String, and UTF16String.

</comment><comment><USER70> I don't understand the reason of using C++ char array to store unicode text....

</comment><comment><USER71> That's because in a truly sane languages there should be a distinction between data type and its implementation.

Then it would be not "string" type, that's broken, but an implementation of "string" type.

<comment><USER72> I agree, it seems like a much saner thing to do. Now that you make me think of that, I do not know many instances of this. I just could think of [https://github.com/clojure-numerics/core.matrix](https://github.com/clojure- numerics/core.matrix) upon which I stumbled recently. Do you have other example of efforts to separate a type from its implementations?

<comment><USER73> doesn't every statically typed imperative language do this, and recommend it?

<comment><USER71> Not to my knowledge.

C++ and Java are statically typed and they, as far as I know, don't have distinction between string interface and implementation, just a standard string type. You can't make your own string implementation and make others (given that - would it exist - they use standard string interface) transparently accept them instead of language's standard string implementation.

Even Haskell (with standard Prelude) doesn't have a readily available and widely accepted typeclass for strings. As String is just an alias to [Char], if library writer used that, they won't accept, say, Data.Text (I know, it's a bit distinct thing, but...)

<comment><USER73> I was referring to "efforts to separate a type from its implementations", not String specifically, and thinking of containers & co.

Although, for example, even java has CharSequence which only gives you access to codepoints in a char sequence, you can inherit from that and create your own.

</comment></comment></comment></comment></comment><comment><USER74> Logically equivalent doesn't mean equivalent for computers. While you can't define why reverse of "noël" is "lëon" by set of rules that computer can follow, computer just can't know.

<comment><USER3> Umm. For that case you definitely can define a valid reversing algorithm. The key is using grapheme clusters as the indivisible base unit. Sure, there are probably some weird languages that will not reverse properly with such algorithm, but it would still be a significant improvement over the current situation.

</comment></comment><comment><USER75> Mostly this article says that most languages choose NFC for their default normalization form, and don't attempt to detect & convert strings to that form automatically.

</comment><comment><USER76> Objective-C handles the "baffle" case just fine.

</comment><comment><USER77> This is now my favorite example of a leaky abstraction.

</comment><comment><USER78> So what do you propose? Not have a string type, and let everyone handle all these cases manually instead? That will not end well...

</comment><comment><USER79> Happy to see that ruby (2.0 at least) passes all the tests except the "baffle" one.

Edit: sadly, it doesn't.

<comment><USER15>
        "noe\u0308l".size # => 5

ruby 2.0.0p247 (2013-06-27 revision 41674) [x86_64-darwin13.0.0]

<comment><USER70> "noe\u{0308}".size => 4

You code is wrong

<comment><USER15> You forgot the "l":

        "noe\u{0308}" # => "noë"

</comment></comment></comment></comment><comment><USER80> It looks like Factor passes all of these tests. Hat tip to Daniel Ehrenberg.

</comment><comment><USER81> Upvotes for the OP who posts "Disrupt 'Is Broken'".

</comment><comment><USER82> What do you guys think about String in Haskell, where it is a list of char? Should it have some other default implementation, or should it have been more, um, decoupled from its implementation (don't know the correct terminology)?

</comment><comment><USER83> Now, look over here! When I substitute this context with that, ka-pow! now it's an array of characters!

Big deal. I don't understand what the point of this article is when it shows the shortcomings of half a dozen different string implementations in random languages. Yes, if you don't understand the language, then your assumptions about how it works may be wrong. Big surprise, that doesn't mean every string implementation needs to conform to your expectations...

<comment><USER84> Unicode is a standard. It says how to act in these circumstances. Calling out incorrect unicode implementations is useful. You shouldn't have to worry about inconsistent behavior between different languages that purport to support unicode strings. That's the point of a standard.

<comment><USER83> So who is the authority about correct unicode implementations, exactly? And how to different languages with different use cases and power conform to such a standard? Why doesn't this authority extend over language implementations? Because they know what they are doing, and understand the domain, unlike the author of this article.

Look, I'm all for open standards, but saying that standards are required to be adhered to at the programming language level is just ignorance of the real world. The point of a standard isn't to dictate how data is architectured internally, it's to facilitate interoperability of systems at their endpoints. If you want interoperability of programmers, than make your own conforming language and get programmers to adopt it the right way, by competing in the market of ideas.

There is no idea here other than the writer's unjustified expectation that he should just know how every language handles Unicode because??? Because Unicode is a standard? No.. that doesn't make sense at all. Mixing contexts to make the point here means there is no ground for his argument to stand on.

<comment><USER84> _So who is the authority about correct unicode implementations, exactly?_

The Unicode Consortium[1] publishes standards. If a language advertises unicode support, I expect it to follow that standard.

 _Look, I 'm all for open standards, but saying that standards are required to be adhered to at the programming language level is just ignorance of the real world_

I'm not saying a language has to do anything, but if it's advertising support for a well defined feature, and does not deliver correctly on that, I will call them out on it, and support anyone else who does as well. Should we all just throw our hands up and say "Well, it's done now, no point in making a big deal of it?" I would rather apply pressure to get things fixed, or at least make it well known enough that future language designers give it the care and attention it's due.

 _There is no idea here other than the writer 's unjustified expectation that he should just know how every language handles Unicode because??? Because Unicode is a standard? No.._

Are you under the impression that what the author is attempting is not well defined? The unicode standard has conformance clauses about how to interpret unicode strings[2]. That means that if a language advertises it has/supports unicode strings, and fails the tests we've just seen, it's not conformant with the unicode standard. That would make this useful because it's pointing out bugs. If a language does _not_ advertise unicode support, but supports _some_ unicode features, then this is useful because it's making sure people are aware of the limits of their language. All too often people refer to the native string implementation in their language as supporting unicode, when clearly there are problems.

1: [http://www.unicode.org/](http://www.unicode.org/)

2: [http://www.unicode.org/versions/Unicode6.2.0/ch03.pdf](http://www.unicode.org/versions/Unicode6.2.0/ch03.pdf) (see section 3.2)

</comment></comment></comment></comment><comment><USER85> Is it possible that Unicode is actually a bunch of horseshit? If literally nobody gets the spec right, then maybe the spec is wrong.

<comment><USER86> Unfortunately, general purpose text is not a clean simple thing that you can model nicely. Unicode is a mess because the problem it tries to solve is messy.

Even if you could somehow come up with something obviously better, getting any new standard adopted widely enough to be useful would be a formidable, if not insurmountable, challenge. It's less pain to keep using Unicode and try to deal with the worst of the damage.

</comment></comment><comment><USER87> I would argue that it's a unicode problem. `U+0308` shouldn't exist in the first place as a unicode character. That's why we have `U+00EB` ('LATIN SMALL LETTER E WITH DIAERESIS'), etc.

<comment><USER29> Not all combination of base and combining characters exist in a precomposed form, since a base character can have an infinite number of combining characters tacked onto it.

If anything should not exist, it's U+00EB, which is a convenience, compatibility and (space) optimisation codepoint.</comment></comment>


The Obligatory WePay "Company Culture" Blog Post - <AUTHOR> http://blog.wepay.com/2010/09/the-obligatory-wepay-%E2%80%9Ccompany-culture%E2%80%9D-blog-post-part-1/ <comment><USER2> You alluded to one danger of ignoring culture: that your company will develop a culture that you don't like. However, there's a bigger concern when you ignore culture - the possibility that your corporation develops multiple cultures, which conflict with one another.

You're already starting to define responsibilities and divide the work accordingly. That's good. However, you need to make sure that there's still a lot of communication between the groups, and not just between the group leaders either. Otherwise what happens is that people get isolated in their own group and begin to think of other groups in the company as the "other". Things only go downhill from there, as these divisions slowly drive the groups into working at cross purposes.

The real job of company culture is to remind employees that they're all in business together; that its not their group against the other groups at the company. If your culture manages that, then it almost doesn't matter what sort of culture it is.

<comment><USER3> At the same time, organizational psychologists seem to recognize that subcultures actually play a vital part of corporations and that they should, generally, not be killed off. They are the watchdogs that keep the dominant culture reality-checked, and they are generally a source of important, but unexpected, new ideas.

A cultish hive-mind where everyone is a clone of the founder is no good either.

It's not an easy balance to strike.

</comment></comment><comment><USER4> 8:30 AM - 9 PM ? Jesus.

<comment><USER5> Don't worry, the next bullet point clearly states they "separated work from personal life."

<comment><USER6> Yes, that's quite easy if the definition of personal life is "Sweet Jesus, sleep!"

More seriously, that's the kind of thing which does creep in to companies. They have a culture already. It's a long hours culture. You may or may not think that's a bad thing, but it's there. It's quite hard to lose as well. People come in, they can't work _less_ than the people already there, so they jump in to it. Soon people that only work 11 hour days are regarded as slackers. Nip it in the bud. It can't work forever, and it'll get harder to change.

<comment><USER7> we don't apologize for working hard. we're a startup trying to make it in a competitive business.

what rich meant was that unlike other companies, where social & work bleed together (people hang at the office 7 days a week, 15 hours a day but aren't always working) - we come in, we jam all day, we go home. and we take weekends off.

<comment><USER8> _we don't apologize for working hard. we're a startup trying to make it in a competitive business._

Past a certain amount of time, you're spinning your wheels, doing bad work you'll have to clean up later, and pushing yourself towards an early burn-out.

It's unhealthy, non-sustainable, a bad cultural precedent.

If there's no other choice, then do it -- we all have. But don't be too proud of it; it's not something you want engrained in your culture.

<comment><USER9> jeeze man, it's 60 hours a week (assuming that we never leave early or take a day off).

It's nowhere near the hours worked by a first year analyst at an ibank, or by a first year associate at a law firm.

Bill is not saying we are particularly proud of how much we work, just that we aren't apologizing for it. Especially when we are talking about the first 5 employees in the company.

Trust me, I don't want to be working 12 hour days a year from now, but it's what the company requires right now. I enjoy coming to work every day and working on something I love. I'd much rather do that for 12 hours a day, than watch the clock for 8.

</comment></comment></comment></comment></comment></comment><comment><USER10> I don't quite know why, but I'm imagining that guy walking in one day and saying, "My name is Tran. Khang Tran. I get stuff done." He sounds like the kind of person I would like to have always behind me, kicking my butt. Also, this was a really well-written blogpost, like a little novel.

<comment><USER9> You nailed it. He has been an inspiration.

</comment></comment><comment><USER11> "When I hear the word 'culture', that's when I reach for my revolver."

</comment><comment><USER12> If I was this Khang guy, I'd be flattered but maybe a bit self-conscious at being so publicly fellated. I mean, really.

</comment><comment><USER13> i found tony hsieh's Delivering Happiness to be a very good read with respect to developing a culture you're proud of.. it's a nice look at what i like to call "tribal ethics." any other recommendations?

<comment><USER9> I just finished reading it, and that's partially what inspired this post (I talk about it in part 2).

This is going to sound ridiculous, but I would read "accidental billionaires" (the book that "the social network" is based on). I think it shows how another type of culture can develop on it's own.</comment></comment>


NoSQL: Distributed and Scalable Non-Relational Database Systems - <AUTHOR> http://www.linux-mag.com/id/7579 <comment><USER2> What bothers me most about the NoSQL movement is that it seems to be in conflict with a central tenet of our community: to avoid premature optimization, especially when at the expense of of expressiveness or correctness.

While normalized relational databases are indeed slower than key/value stores, they also foster data integrity and the ability to compose arbitrarily complex queries with little effort (assuming one actually groks SQL). One can always optimize trouble spots in a relational database later on (possible even with a KVS), but using a KVS from the get-go is bound to lead to pain.

If we happily use Ruby, Python, et al. while chanting the mantra of premature optimization being bad (and rightly so!), why does this not apply also to databases?

<comment><USER3> That's like saying a Formula 1 team should first design a Ford Focus and then optimize it to win the world championship. It just doesn't make sense to call that process an optimization. The same is true for hi-performance/hi-volume use-cases where KV stores make sense.

Also, putting the optimization issue aside, the KV data access model is actually quite natural for web applications. This is hard to believe (seriously), and you have to see it to believe it. Note that in web apps, being able for the administrator to go in and issue complex queries/updates is much less important than for eg. a billing system. I use an RDBMS for an old billing project, and it wouldn't make sense to use a KV store, but I use a KV store for a web app I'm writing, and it turns out to be a great fit, more natural than an RDBMS.

<comment><USER2> Sure, I suppose it matters a lot on the type of Web application you're building. If you have a relatively simple domain model with very few relationships, I'm sure a KVS works pretty well. Still, for anything other than applications with simple domain models, or ones that need tremendous performance, a KVS is inappropriate outside of isolated optimization -- in practice, one runs into headaches with non-normalized data very quickly (think MBAs and their giant, messy Excel spreadsheet).

<comment><USER3> I disagree.

In most web apps data in per-user (or per-object, whatever that object is), and it's not a major headache to denormalize it to be per-user. One notable is exception is a social network, but that can be handled with some care.

We're talking about web apps written by programmers, not MBAs. Sure, if your web apps are written by MBAs, then they should use an RDBMS, because they can do less damage (?).

I'm not arguing against RDBMS/SQL here, all I'm saying is that a KV store is actually a good choice for the domain of web applications. You can also use an RDBMS of course, as we have for the last 10 years with the LAMP stack, as long as you're handling a managable amount of traffic. Even with a lot of traffic you can handle it with an RDBMS, it just stops making sense at some point.

</comment></comment></comment></comment><comment><USER4> Weird, the only distributed system that gets even a passing mention is Voldemort.

</comment><comment><USER5> Naturally, Hypertable and HBase, two of the more prominent and technologically advanced solutions aren't even included in this article.

<comment><USER6> He probably equates NoSQL to KV stores that has no chance of having SQL implemented, as they have no efficient range scan capabilities. It's totally feasible to have full SQL (and more) implementation on top over Hypertable.

<comment><USER4> > It's totally feasible to have full SQL (and more) implementation on top over > Hypertable.

Except that would be ridiculous, since the data model is just not designed for denormalization. (And that's okay!)

<comment><USER6> Denormalization can happen automatically based on the usage of queries with joins. Some commercial RDBMS like DB2 already do that. SQL is a declarative language. It doesn't force you to use a particular implementation.

<comment><USER4> That doesn't make it a good idea. :)

It's such a bad fit because your keys will be spread across other nodes in the cluster. Even appengine datastore, which doesn't go nearly all the way towards full sql and still emphasizes denormalization, has latency commonly in 100s of ms.

<comment><USER6> You're guessing the implementation here. It's only a bad idea if you're doing it wrong :) The source/conceptual tables can be normalized, while the join views can be materialized/implemented with Bigtable like sparse columns, which are dynamically updated if the joins happen a lot. Any subsequent join queries hits the views but the source tables. Manual denormalization should be allowed but not required.

GAE has nothing to do with this, it's a shared cluster, such that any benchmark is not reliable and comparable. It also has nothing to do with the SQL discussion, as it doesn't support joins either.</comment></comment></comment></comment></comment></comment>


Atlassian UI opensourced - <AUTHOR> http://confluence.atlassian.com/display/AUI/Atlassian+User+Interface+(AUI)+Developer+Documentation

<comment><USER2> Am I missing something? It appears to be only an API developers so their Atlassian plugins have a consistent look and feel. I also cannot find any source nor license over the AUI components.

</comment><comment><USER3> Erm. Another JS framework. Seriously?

C'mon guys, everyone is using jquery now. And YUI will also be around for a while longer, until everyone has finally recovered from their bad java childhood.

What does this bring to the table that neither those two elephants nor one of the 28(!) other js libraries have?

<comment><USER4> This is a set of UI components, not a general purpose framework like jQuery. The two libraries have almost nothing in common.

Separately, I don't agree with your premise that more frameworks is a Bad Thing, nor that jQuery is/should be used in all projects.</comment></comment>


No One Forced Me to Drop Bitcoin Trademark Application, Lawyer Says - <AUTHOR> http://www.betabeat.com/2011/07/08/no-one-forced-me-to-drop-bitcoin-trademark-application-lawyer-says/

<comment><USER2> Mr "Pascazi" - refer to <http://bit.ly/pEgrbE>.

Notably 5.8 in this case - i'll keep a watch but if anyone notices a filing in Australia - let me know and i'm happy to write and object :) Also, incase this fool didn't realize - the evidence trail he is leaving across the web implying that he intends to use it in bad faith is well ... amateur at best.

Also - perhaps Pascazi needs to understand technology a bit more. Notably, learning how bitcoin works.

Let's Pascazi a lesson in law and take him ..... _Puts on sunglasses_ ..... to bits .... YEEEAAAHHHHHHHHH

<comment><USER3> In case anyone else is mistrustful of shortened links, that one leads to a page entitled "Grounds for opposition to registration of national trade marks" on www.ipaustralia.gov.au

</comment></comment><comment><USER4> The name doesn't matter, the technology is what makes Bitcoin. We could change the name to BitBills and I don't think anyone would miss a beat, the work would move forward and those that matter would move along with it.

<comment><USER5> True, but that doesn't change the fact that it will be a total perversion of the fundamental purpose of trademark law if this guy can manage to register a name that _other people are already using_ , whether or not they've registered it.

Trademark law is for helping you protect a business you're building, not to give you a way to steal from others who have already made a mark significant.

I hope this guy wastes a bunch of his own money and gets nowhere.

</comment></comment><comment><USER6> SO we can trademark the name pascazi?

Now that would be ironic</comment>


Facebook Will Go Public or Disclose Financials by April 2012  - <AUTHOR> http://online.wsj.com/article/SB10001424052748703730704576066162770600234.html?mod=WSJ_hp_LEFTTopStories

<comment><USER2> Interesting that the goldman clients have to agree to not sell their shares until 2013, so once they finally know the full financials they'll still be prohibited from selling for almost a year.

<comment><USER3> An agreement without a penalty is meaningless.

There is no way for goldman to enforce those agreements.

And if nothing else they can simultaneously short and hold the stock if they really want to sell it, but still stay within the letter of the agreement.

<comment><USER2> Pretty sure goldman will have no trouble enforcing the contract considering if you want to sell the SPV shares you'll have to do it through goldman.

</comment></comment></comment><comment><USER4> On Intrade, would you bet _for_ or _against_ Facebook IPO'ing by April 2012?

People don't seem to be putting their money where their mouths are. Lack of confidence?

</comment><comment><AUTHOR> Interesting that the WSJ changed the title of the article. It now reads: "Facebook Opens Door for Financial Disclosure or IPO in 2012"

</comment><comment><USER5> How come this is such a newsworthy topic? "Big company with one of the world's most popular products might be going public." Well, yeah.

If you care, save your money and buy their stock when it becomes available. But do you really need to write an article?

Tune in at 11: The Sky is Blue.

<comment><USER6> Considering they've avoided breaking the 500 investor mark to this point so they would not have to disclose financials (hence the whole Goldman Sachs silliness) this IS news worthy.</comment></comment>


BitPay, PayPal for Bitcoin, raises $510K – already has 2,100 businesses on board - <AUTHOR> http://techcrunch.com/2013/01/07/bitpay-banks-500k-in-angel-investment-to-become-paypal-for-bitcoin-already-has-2100-businesses-on-board/ <comment><USER3> Nice idea. I've gone through their website and even watched the explanation video.

One thing they don't mention is how buyers come into possession of bitcoins. Buying bitcoins is currently a hassle for an average consumer and has transactional fees that are paid to bitcoin exchanges - enterprises of unknown origin and reputation that don't have household level brand name recognition (ask your aunt Sally or uncle Kamal about Mt. Gox).

I assume that chargebacks that are offloaded to bitcoin exchanges. These exchanges are the weakest point of the bitcoin ecosystem at the moment - <USER2> is where bitcoins touch the real world.

Contact with the real world makes them susceptible to the control of the authorities (regulators, law enforcement, criminals and etc), easy target for malicious parties meaning harm (yes, don't want to use the word hackers) as well as financial manipulators.

Success of BitPay and the like depends on the health and stability of the whole ecosystem, which is still quite immature, so if I were a merchant I'd approach this cautiously.

<comment><USER4> Check out <http://howdoyoubuybitcoins.com/>

I recommend Coinbase personally, it's very easy! Come buy a cupcake at Cups and Cakes in SF too :)

<comment><USER5> I've never understood the cost to buying bitcoins to begin with. You need to use a credit card I imagine, which will say have a 2.9%+ fee associated with it?

<comment><USER4> One word: Chargebacks.

If someone decides 60 days later that they want to contest the purchase, they can. Usually this screws over the seller, leaving them in the negative.

<comment><USER6> I am not sure I grasp your point entirely. Could you rephrase or reiterate this idea?

</comment></comment></comment></comment></comment><comment><USER7> I've got other projects filling my time, but I think there's a great opportunity for someone who isn't opposed to working on an adult-themed idea using bitcoin. Selling sex toys for bitcoin, specifically. I think the anonymity of it is a huge selling point. Yes, there is still shipping to be concerned with, but in exchange for no worries about line items on credit card statements or your neighbors seeing your car at the sex shop; it might be the lesser of all evils.

If you do this and get rich, please email me to tell me it worked!

<comment><USER8> Just playing devils advocate... the only person who sees my CC statement other than me is my girlfriend. So I don't really care what is on it as long as it isn't causing a domestic problem... :-)

Using your logic that you could use Bitcoin to hide things on your CC statement I am guessing the use case for most people is hiding things from their spouses.

<comment><USER9> Playing devil's advocate re CC bills, one of my friends has always maintained that the only reason he hasn't paid for porn online is because he doesn't want his bank manager to know he's gay. (Regardless of what would appear on his bank statement, it's just not something he wanted to risk - I've never asked for an in depth reason as to why it's a risk).

<comment><USER10> I worked in IT at a bank. No one ever just reads account details for fun. If you call about a question on your account then maybe they will. But by and large employees really don't care at all. Besides, most transaction names are pretty cryptic, especially from sex sites, who often use a 3rd party service.

<comment><USER11> Former bank teller here --

I've seen it happen where a couple would be sitting down together working out some account fraud with one of the branch managers and some dubious charges would come up. In my four years working there, saw probably 3-4 couples storm out in anger related to something the other found on the account related to pornography

<comment><USER12> I get that this is, in fact, a real problem - but it's hardly one related to banking or finance. Sounds more like relationship trouble to me.

I'd be more concerned about the bitcoin conversion services showing up in some sort of record than porn since it's so easy to connect with far more questionable services. Yes, a lot of people want to be off the grid (especially financially), but the reality is that a lot of people buy into the "why do you need privacy if you have nothing to hide" mantra. Even spouses.

<comment><USER7> When I bought some bitcoins, I did it with cash. I took money to a bank teller, and wired it to an account; then the service sent me the bitcoin via email. I never showed ID, so there was no way to track it to me. I suppose if someone cared enough to track a cash withdrawal they could do that, but in no way was my bank account connected with a bitcoin transaction at any time.

</comment></comment></comment></comment></comment></comment></comment><comment><USER4> If you're interested in buying bitcoins, check this out:

<http://howdoyoubuybitcoins.com/>

We have regional guides, as well as company reviews+ratings so you can buy with trust.

<comment><USER13> How many times are you going to spam this discussion with that URL? I count 3 so far.

<comment><USER14> Apparently it's necessary, given the number of "Nobody knows how to buy bitcoin" posts.

</comment></comment></comment><comment><USER8> Until there is an easy mainstream way to fund a bitcoin wallet then I think bitcoin payment gateways are kind of pointless and will struggle to get any real adoption.

<comment><USER15> Coinbase has been discussed several times on HN.

<comment><USER16> Hooking up a bank account to a bitcoin related startup seems risky, and waiting 4 days for funds to post seems inconvenient.

<comment><USER17> Why do you think it's risky? In principle, is it that different than using Facebook credits? I don't think Bitcoin is illegal in US.

<comment><USER18> > Why do you think it's risky?

Probably because every new bitcoin processor seems to rediscover the same set of security holes and insecure practices.

<comment><USER19> Coinbase is run by Brian Armstrong who used to lead anti-fraud at Airbnb. He's a pro.

</comment></comment></comment></comment></comment></comment><comment> shawn-butler I don't see any evidence on the web or in that airy press release other than its headline to support that they have 2,100 real world businesses accepting payments.

Does anyone have the list? I'm not trying to be negative it's just that I would like to have some basis in fact before I inadvertently spread someone else's mistake in my advocacy!

</comment><comment><USER4> For those who are interested, Cups and Cakes Bakery in San Francisco uses Bitpay to allow customers to buy cupcakes with Bitcoins instantly. Here's an article and video showing what it's like:

[http://cupsandcakesbakery.com/2012/10/buy-cupcakes-with- bitc...](http://cupsandcakesbakery.com/2012/10/buy-cupcakes-with-bitcoins/)

</comment><comment><USER20> The more mainstream Bitcoin becomes, the greater the likelihood that it will be shut down. As an alternate currency, it's a direct threat to the dollar. That threat is miniscule at present, but as Bitcoin gains traction, through services like Bitpay that reduce friction, it will be targeted for elimination by the economic powers that be.

<comment><USER21> Yep. But then the question is how will they shut it down?

The powers that be hate bittorrent, but it's still alive and prospering.

The powers that be hate drugs, but the war on drugs has been an abysmal failure.

Bitcoin will prove to be very difficult for the powers that be to shut down. That is why it has so many fervent early adopters.

<comment><USER20> I suspect a number of strategies will be used. First, they'll say it's being used for drug transactions. (Which is of course already the case.) Then they'll say it's being used for terrorist funding.

Once it's thoroughly demonized, they'll start going after people with bitcoin balances, and seize them through existing asset forfeiture laws. (Which allow the government to seize whatever they want; and if you want it back, the burden of proof is on you.)

They'll probably also focus on methods for converting Bitcoin back into mainstream currencies. In the end, those massive NSA databases that record everything that everyone is doing will probably be used to crack down on larger numbers of individuals. While this wouldn't be a "technical" shut-down, it would have a chilling effect as these "examples" will be publicized as a warning to other Bitcoin users.

<comment><USER22> > Once it's thoroughly demonized, they'll start going after people with > bitcoin balances

How do you determine that someone has bitcoin balance?

> They'll probably also focus on methods for converting Bitcoin back into > mainstream currencies.

This sounds more viable strategy: attack the exchanges. After all, the value of bitcoins comes from the fact that it is liquid - easily convertible to existing currencies and back.

However even shutting the exchanges down would require global cooperation from goverments and probably heavy internet censorship as well. That is because many exchanges don't even need any ties to the existing banking infrastructure, and can operate from any country.

</comment></comment></comment></comment><comment><USER23> Isn't it a risky move to build a company around a cryptographic concept that wasn't around for that much time ?

<comment><USER24> Yes, but often times the size of the payoff is directly proportional to the amount of risk involved.

</comment></comment><comment><USER25> "PayPal for ______" is the last thing I'd want my company associated with ever, for any reason, period.

</comment><comment><USER26> I'm sure this is on their website, but:

They accept bitcoins, they pay in local currency. Where are they acquiring their local currency? Are they:

\- Selling bit coins to consumers?

\- Selling bit coins on open exchanges?

\- - If they are selling on open exchanges, are all the local currencies currently available? I'd guess not, so they must have currency reserves in all outbound countries?

\- Paying their staff in bit coins? :-D

This would be a _huge_ multi-national operation. Not least in running the bank accounts in outbound countries, and managing ForEx reserves and bids. Let alone dealing with money laundering laws in respective countries, commercial operations expatriating or repatriating profits/revenues, etc. etc. Not for me!

</comment><comment><USER17> Is Bitpay the "Paypal for Bitcoin" only from a merchant point of view, or from a user's point of view, too? I haven't looked at it much, but I think their web page was more oriented towards merchants not users, while Coinbase seems to be more oriented towards users, and sending money to each other and whatnot.

<comment><USER4> I really think Coinbase is the Paypal for bitcoin. Paypal should be worried... they are growing VERY quickly.

<http://howdoyoubuybitcoins.com/from/coinbase/>

</comment></comment><comment><USER27> Bitcoin will be at $20 by the end of 2013. Great move on the part of all the principals involved in this deal.

<comment><USER28> I don't see any reason for BitPay to be long bitcoin. If you're a payment provider currency volatility is bad both ways.

<comment><USER29> BitPay is essentially bitcoin arbitrage. A buyer wants to purchase something with bitcoins. A seller wants to receive payment in USD (or EUR, etc). BitPay buys the bitcoins and gives the seller conventional currency. They're left hanging on to bitcoins. I don't know what they do with the bitcoins once they have them, but I bet they're sitting on them and waiting for their value to increase.

<comment><USER15> If you want to go long Bitcoin, just buying them would probably be easier. And there is some evidence that BitPay sells off their Bitcoins: <https://bitcointalk.org/index.php?topic=89757.0>

<comment><USER29> Buying them outright would be easier, but you wouldn't get the 1% transaction fee that BitPay enjoys.

Thanks for the link. It sounds like they buy more bitcoins than they can sell off without crashing the market, so they're probably holding on to most of it. Interesting risks there.

</comment></comment></comment></comment></comment><comment><USER8> Yeah I am Canadian so neither Coinbase or BitInstant work for me. As far as I can tell my best option is adding a Bitcoin payment site as a "Bill Payee" in my online banking and paying it like a bill. Not exactly mainstream... :-)

Even BitInstant and Coinbase are kind of clunky and require the user to have an understanding of how Bitcoins work. (e.g. understand wallets, etc). That is the nice thing about Paypal...you can either login and pay with your paypal account or use them as a CC processor if you don't have an account.

<comment><USER30> As a Canadian, you can walk in any branch of 3 banks and deposit cash to get Bitcoins through Canadian Virtual Exchange, see <https://www.cavirtex.com/methods>

</comment></comment><comment><USER31> When I read that headline I ask myself is that $510k _real_ money or is it the Bitcoin-equivalent of that sum?

And the fact that I ask that, as a techie - an enthusiast in the know about stuff like this, probably means Bitcoin still has some way to go before it can hope to achieve any sort of mainstream success.

<comment><USER32> You weren't sure if they were talking cash or cash equivalent, and therefore concluded that Bitcoin still has a way to go?

I'm confused, because to me that reads as "I allowed for the possibility of this to have all been done in BTC, and therefore BTC still has a long way to go..."

</comment></comment><comment><USER33> Very glad to hear this. I went with BitPay about a month ago and the Bitcoin orders are starting to roll in now that the community has gotten wind. Anyone looking to expand their audience should consider offering Bitcoin payments, regardless of which processor you go with (BitPay, CoinBase, etc.).

</comment><comment><USER34> I was really surprised they were asking for my and my company's personal information during the registration process. Bitcoin is supposed to anonymous. So I went with Coinbase instead, they don't do that.

<comment><USER27> Coinbase wants a link to a bank account in order to buy bitcoins. How is that less intrusive?

<comment><USER34> Well, not for me. I only want to be able to _receive_ payments as a merchant. At the moment I don't care about exchanging bitcoins and bank accounts. BitPay, on the other hand, wants all the information _before_ I can start receiving payments using their gateway.

<comment><USER27> Ah. I see what you're saying now. Yes, Coinbase has an easier entry than Bitpay does.

</comment></comment></comment></comment><comment><USER35> Maybe obvious but you can get most of this benefit with Liberty Reserve. Non- reversible transactions and you can fill your liberty reserve account with bitcoins.

</comment><comment><USER36> I don't get this. Isn't the whole point of Bitcoin not to have any "central" entity? BitPay sounds like precisely this?

<comment><USER15> There's nothing particularly central about it. It's a payment intermediary for merchants (I would say it's more like Stripe than Paypal). Nobody has to use it.

</comment></comment><comment><USER37> This is 1/8 the money that Cherry (joke of 2012) raised. Is it becoming harder to fund ideas?</comment>