The Python Unicode Mess - <AUTHOR> http://changelog.complete.org/archives/9938-the-python-unicode-mess <comment><USER4> As far as I can tell this is a long-form "I used to be able to ignore encoding issues and now it's a 'mess' because the language is forcing me to be correct". Each of the examples cited is something which was a source of latent bugs which he thought was working because they were ignored.

Only his third bit of advice isn't wrong and treating it as something unusual shows the problem: the only safe way to handle text has always been to decode bytes as soon as you get them, work with Unicode, and then encode it when you send them out. Anything else is extremely hard to get right, even if many English-native programmers were used to being able to delay learning why for long periods of time.

<comment><USER5> > As far as I can tell this is a long-form "I used to be able to ignore > encoding issues and now it's a 'mess' because the language is forcing me to > be correct".

The problem with that view is that there are things for which you _can not_ be correct, and there are no encoding issues because there _is no encoding_ (or if there is one it does not map to proper unicode):

* UNIX files and paths have no encoding, they're just bags of bytes, with specific bytes (not codepoints, not characters, bytes) having specific meaning

* Windows file and path names are sequences of UTF-16 _code units_ but not actually UTF-16 (they can and do contain unpaired surrogates), as above with specific code units (again not codepoints or characters) having specific meaning

These are issues you will encounter on user systems, there is no "forcing you to be correct". A non-unicode path is not incorrect. On many systems it just is. OSX is one of the few systems where a non-unicode path is actually incorrect, and that means you will not encounter one as input so you have no reason to handle this issue at all.

> Only his third bit of advice isn't wrong and treating it as something > unusual shows the problem: the only safe way to handle text

That's where you fail, and to an extent so does Python: some text-like things _are not actually text_. Path names famously is one of this case. You're trying to hammer the square peg of path names in the round hole of unicode.

<comment><USER4> That's just restating my point: Unix filenames are bytes (on most filesystems, anyway). The fact that many people were able to conflate them with text strings was a convenient fiction. Python no longer allows you to maintain that pretense but it's easy to deal with by treating them as opaque blobs, attempt to decode and handle errors, or perform manipulations as bytes.

<comment><USER6> One thing that amuses me given the number of complaints about the Python 3 string transition is how vastly _better_ Python 3 is for working with bytes. The infrastructure available is light-years ahead of what Python 2 offered, precisely because it gave up on trying to also make bags of bytes be the default string type.

<comment><USER7> Thank you for saying that. Working with strings and bytes in Python 3 is nothing short of a joy compared to the dodgy stuff Python 2 did. People who complain about the change are delusional.

<comment><USER8> The only problem I have with Python3 strings/bytes handling is the fact that there are standard library functions which accept bytestrings in Py2 (regular "" strings), and Unicode strings in Py3 (again, regular "" strings in Py3).

This has led to developers attempting to conflate the two distinctly different concepts and make APIs support both while behaving differently.

A simple solution is there in plain sight: just use exclusively b"" and u"" strings for any code you wish to work in both Py2 and Py3, and forget about "". All and any libraries should be using those exclusively if they support both. Python3-only code should be using b"" and "" instead.

One could consider this a design oversight in Python 3: the fact that the syntax is so similar elsewhere makes people want to run the same code in both, yet a core type is basically incompatible.

<comment><USER9> u"" is a syntax error in python3 (or at least it was for a while, apparently its not anymore, that said...). The correct cross-platform solution is to do

        from __future__ import unicode_literals

which makes python2 string literals unicode unless declared bytes. Then "" strings are _always_ unicode and b"" strings are always bytes, no matter the language version.

<comment><USER7> > u"" is a syntax error in python3

This has not been the case since 2012. The last release of Python 3 for which this was the case reached end of life in February 2016. Please stop misinforming people.

<comment><USER10> While u"" is accepted in current Python 3, for some reason they ignored the raw unicode string ur"" which still is a syntax error in Python 3. So, unicode_literals is definitely preferable.

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER11> Text encoding in general is a mess, and Python 2 Unicode support was a mess, but Python 3 makes it _much less_ of a mess.

I think the author has a mess on his hands because he's trying to do it the Python 2 way – processing text without a known encoding, which is not really possible, if you want the results to come out right.

To resolve the mess in Python 3, choose what you actually want to do:

1\. Handle raw bytes without interpreting them as text – just use bytes in this case, without decoding.

2\. Handle text with a known encoding – find out the encoding out-of-band from some piece of metadata, decode as early as possible, handle the text as strings.

3\. Handle Unix filenames or other byte sequences that are usually strings but could contain arbitrary byte values that are invalid in the chosen encoding – use the "surrogateescape" error handler; see PEP 383: [https://www.python.org/dev/peps/pep-0383/](https://www.python.org/dev/peps/pep-0383/)

4\. Handle text with unknown encoding – not possible; try to turn this case into one of the other cases.

Also, watch Ned Batchelder's excellent talk, _Pragmatic Unicode, or, How do I stop the pain?_ , from 2012: [https://pyvideo.org/pycon-us-2012/pragmatic- unicode-or-how-d...](https://pyvideo.org/pycon-us-2012/pragmatic-unicode-or- how-do-i-stop-the-pain.html)

<comment><USER12> > To resolve the mess in Python 3, choose what you actually want to do...

The thing is that this is not actually going to happen. Programs are simply broken _across the board_ , because few people can be bothered to deal with all these peculiarities.

The difference is, in Python 2, output would be corrupted in some edge cases, but generally it would "just work". In Python 3, the program falls flat on its face even in cases that would've ended up working fine in Python 2.

I don't think there's a general answer on which behavior causes less real- world problems total, but the idea that Python 3 makes less of a mess is not something I can agree with.

<comment><USER13> This just reflects your experience of only speaking english, to most of the world their native language is _not_ and edge case.

<comment><USER12> > This just reflects your experience of only speaking english, to most of the > world their native language is not and edge case.

Excuse me, I don't exclusively deal in 7-bit ASCII characters just because I happen to speak English, which isn't the _only_ language I speak either.

</comment></comment></comment></comment><comment><USER14> There is a particular use case which leads to frustration with Python 3, if you don't know the latin1 trick.

The use case is when you have to deal with files that are encoded in some unknown ASCII-compatible encoding. That is, you know that bytes with values 0–127 are compatible with ASCII, but you know nothing whatsoever about bytes with values 128–255.

The use case arises when you have files produced by legacy software where you don't know what the encoding is, but you want to process embedded ASCII- compatible parts of the file as if they were text, but pass the other parts (which you don't understand) through unchanged (for example, the files are documents in some markup language, and you want to make automatic edits to the markup but leave the rest of the text unchanged). Processing as text requires you to decode it, but you can't decode as 'ascii' because there are high-bit- set characters too.

The trick is to decode as latin1 on input, process the ASCII-compatible text, and encode as latin1 on output. The latin1 character set has a code point for every byte value, and bytes with the high bit set will pass through unchanged. So even if the file was actually utf-8 (say), it still works to decode and encode it as latin1, and multi-byte characters will survive this process.

The latin1 trick deserves to be better known, perhaps even a mention in the porting guide.

<comment><USER15> > _The latin1 character set has a code point for every byte value_

No it doesn't. The whole range of 128-159 are undefined. However, the old MS- DOS CP-437 encoding (which is incompatible with latin1/ISO-8859-1) _does_. So your trick is valid, but _not_ with latin1.

<comment><USER15> I can't edit my post now, but it turns out I was _wrong_. The range of 128-159 _are_ defined in ISO-8859-1, as little-used "control characters":

[https://en.wikipedia.org/wiki/C0_and_C1_control_codes#C1_set](https://en.wikipedia.org/wiki/C0_and_C1_control_codes#C1_set)

So, the trick described by garethrees does work with latin1, and I was mistaken.

</comment></comment></comment><comment><USER16> The real problem here is that

* UNIX file systems allow any byte sequence that doesn't contain / or \0 as file and directory names

* User interfaces have to render that as strings, so they must decode

* There is no meta data about what the file encoding is

Many programs use the encoding from the current locale, which is mostly a good assumption, but the way that locales scope (basically per process) has nothing to do with how file names are scoped.

So, many programs make some assumptions. Some models are:

1) Assume file names are encoded in the current locale

2) Assume file names are encoded in UTF-8

3) Don't assume anything

The "correct" model would be 3), but it's not very useful. People want to be able to sort and display file names, which generally isn't very useful with binary data.

Which is why most programs, including python, use 1) or 2), and sometimes offer some kind of kludge for when the assumption doesn't hold -- and sometimes not.

IMHO a file system should store an encoding for the file names contained in it, and validate on writes that the names are correct. But of course that would be a huge POSIX incompatibility, and thus won't happen.

People just live with the current models, because they tend to be good enough. Mostly.

<comment><USER17> Would it really be POSIX-incompatible? Does the standard mandate that a filesystem can place no such restriction on top of "filenames are unencoded bytes"? If not, then it's just that tools cannot blindly assume filenames are decodable. Isn't MacOS guaranteeing UTF8 these days, while still being POSIX- compliant?

<comment><USER16> I don't know if there is an explicit mandate for that in the standard, but forbidding this that were previously allowed, both in code and documentation, is not backwards compatible.

Just imagine a file system that wouldn't allow the character "e" in file names.

Of course, the impact would be not as drastic, but still it's backwards incompatible.

</comment></comment></comment><comment><USER18> I'm not so sure other languages do that any better (nodejs doesn't even support non-unicode filenames at all for instance). Modern python does a pretty good job at supporting unicode, very far away from being a "Mess" that's just very much not true at all. People always like to hate on python but then other languages supposedly designed by actually capable people do mess up other stuff all the time. Look at how the great Haskell represents strings for instance and what a clusterfuck[1] that is.

[1] [https://mmhaskell.com/blog/2017/5/15/untangling-haskells- str...](https://mmhaskell.com/blog/2017/5/15/untangling-haskells-strings)

<comment><USER5> Rust is probably one of the languages which does this crap best, and that's thanks to static typing and deciding to not decide:

1\. it has proper, validated _unicode_ strings (though the stdlib is not grapheme-aware so manipulating these strings is not ideal)

2\. it has proper _bytes_ , entirely separate from strings

3\. it has "the OS layer is a giant pile of shit" OsString, because file paths might be random bag of bytes (UNIX) or random bags of 16-bit values (and possibly some other hare-brained scheme on other platforms but I don't believe rust supports other osstrings currently)

4\. and it has nul-terminated bag o'bytes CString

For the latter two, conversion to a "proper" language string is explicitly known to be lossy, and the developer has to decide what to do in that case for their application.

<comment><USER19> > 1\. it has proper, validated unicode strings (though the stdlib is not > grapheme-aware so manipulating these strings is not ideal)

Grapheme clusters are overrated in their importance for processing. The list of times you want to iterate over grapheme clusters:

1\. You want to figure out where to position the cursor when you hit left or right.

2\. You want to reverse a string. (When was the last time you wanted to do that?)

The list of times when you want to iterate over Unicode codepoints:

1\. When you're implementing collation, grapheme cluster searching, case modification, normalization, line breaking, word breaking, or any other Unicode algorithm.

2\. When you're trying to break text into separate RFC 2047 encoded-words.

3\. When you're trying to display the fonts for a Unicode string.

4\. When you're trying to convert between charsets.

Cases where neither is appropriate:

1\. When you want to break text to separate lines on the screen.

2\. When you want to implement basic hashing/equality checks.

(I'm not sure where "cut the string down to 5 characters because we're out of display room" falls in this list. I suspect the actual answer is "wrong question, think about the problem differently").

Grapheme clusters is relatively expensive to compute, and its utility is very circumscribed. Iterating over Unicode codepoints is much more useful and foundational and yet still very cheap.

<comment><USER20> > Grapheme clusters are overrated in their importance for processing. The list > of times you want to iterate over grapheme clusters:

> 1\. You want to figure out where to position the cursor when you hit left or > right.

> 2\. You want to reverse a string. (When was the last time you wanted to do > that?)

You missed the big one:

3\. You want to determine the logical (and often visual) length of a string.

Sure, there are some languages where logical-length is less meaningful as a concept, but there are many, many languages in which it's a useful concept, and can only be easily derived by iterating grapheme clusters.

<comment><USER21> Visual length of a string is measured in pixels and millimetres, not characters. In a font/graphics library, not in a text processing one.

<comment><USER20> Sorry, visual length as in visual number of "character-equivalent for purposes of word length" things. Those things are close to, but not exactly the same as, grapheme clusters, so the latter can often be used as an imperfect (but much more useful than unicode points or bytes) proxy for the former.

There's no perfect representation of number-of-character-equivalents that doesn't require understanding of the language being handled (and it's meaningless in some languages as I said), but there are _many_ written languages in which knowing the length in those terms is both extremely useful and extremely hard to do without grapheme cluster identification.

<comment><USER22> > _character-equivalent for purposes of word length_

Serious question: why would you want to do this?

I know it's fashionable to limit usernames to X characters... but why? The main reason I've seen has been to limit the _rendered length_ so there are some mostly-reliable UI patterns that don't need to worry about overflows or multiple lines. At least until someone names themselves:

W W W W W W W W W W W W W W W W W W W W

Which is 20 characters, no spaces, and will break loads of things.

(I'm intentionally ignoring "db column size" because that depends on your encoding, so it's unrelated to graphemes)

<comment><USER6> _Serious question: why would you want to do this?_

Have you never, in your entire life, encountered a string data type with a length rule? All sorts of ID values (to take an obvious example) either have fixed length, or a set of fixed lengths such that every valid value is one of those lengths, and many are alphanumeric, meaning you cannot get round length checks by trying to treat them as integers. Validating/understanding these values also often requires identifying what code point, not what grapheme, is at a specific index.

Plus there are things like parsing algorithms for standard formats. To take another example: you know how people sometimes repost the Stack Overflow question asking why "chucknorris" turns into a reddish color when used as a CSS color value? HTML5 provides an algorithm for parsing a (string) color declaration and turning it into a 24-bit RGB color value. That algorithm requires, at times, checking the length _in code points_ of the string, and identifying the values of code points at specific indices. A language which forbids those operations cannot implement the HTML5 color parsing algorithm (through string handling; you'd instead have to do something like turn the string into a sequence of ints corresponding to the code points, and then manually manage everything, and why do that to yourself?).

<comment><USER22> Yes. All instances I've seen have been due to byte-size restrictions (so it depends on encoding) or for visual reasons (based on fundamentally flawed assumptions). With exceptions for dubious science around word-lengths between languages / as a difficulty/intelligence proxy, or just having fun identifying patterns. (interesting, absolutely, but of questionable utility / bias)

But every example you've given have been around visuals, byte sizes, or code points (which are unambiguously useful, yes). Nothing about _graphemes_.

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER23> My main criticism of Python 3's changes to strings is that it has become much more specific about strings.

In Python 2, if you have a series of bytes -or- a "string", the language has no opinion about the encoding. It just passes around the bytes. If that set of bytes enters and exits Python without being changed, its format is of no concern. Interactions do not force you to define an encoding. This is _not correct_ , but it is _often functional._

Python 3, on the other hand, if you ever treat bytes as a string, forces you to have an opinion about the encoding. Same goes for if you convert back to bytes. For uncommon or unexpected encodings, the chance of this going wrong in a casual, accidental way is much higher. Of course, the approach is more correct, but it doesn't _feel_ more correct to the programmer.

<comment><USER7> > it doesn't feel more correct to the programmer.

I agree with the details of what you said, but the insidious thing about how Python 2 organized strings and encodings is that most programmers were free to ignore it and produce buggy software. Then, later, people who had to use that software on non-ascii data would try to use it and it would blow up. This would lead to a very painful cycle of shaking out bugs that the original author may not even be motivated to fix.

The decision to force encodings to be explicit and strings/bytes to be separate was a great design change. It literally made all our code more valuable by removing hidden bugs from it.

</comment></comment><comment><USER24> For anyone interested in learning why Python 3 works this way I highly recommend the blog of Victor Stinner[0].

As for the article, this is nothing new. The problem is similar to the issues raised by Armin Ronacher[1]. These problems are well known and Python developers address them one at a time. Issues around these egde cases have improved since the initial release of Python 3.0.

[0] [http://vstinner.github.io](http://vstinner.github.io)

[1] [http://lucumr.pocoo.org/2014/5/12/everything-about- unicode/](http://lucumr.pocoo.org/2014/5/12/everything-about-unicode/)

</comment><comment><USER25> This article is kind of hard to evaluate, because the OP doesn't provide an example program with an example input that fails. So it's hard to judge whether the solution presented here is actually ideal. Instead, we're forced to just take the OP's word for it, which is kind of uncomfortable.

I do somewhat agree with the general sentiment, although I find it difficult to distinguish between the problems specifically related to its handling of Unicode and the fact that the language is unityped, which makes a lot of really subtle things very implicit.

<comment><USER26> The OP links to StackOverflow, where failing inputs are mentioned in the comments on the accepted answer. And the second-most upvoted answer explains that _.decode( 'unicode_escape')_ only works for Latin-1 encoded text: [https://stackoverflow.com/a/24519338](https://stackoverflow.com/a/24519338)

<comment><USER27> The question being how to parse character escapes (backslash sequences) in Python.

To be honest, you could write a custom character-by-character parser easily or even use the regex module.

</comment></comment></comment><comment><USER28> IMHO the whole python3 string mess could have been prevented if they had chosen UTF-8 as the only string encoding instead of adding a strict string type with a lot of under-the-hood magic. That way strings and byte streams could remain the same underlying data, just as in python2. The main problem I have with byte-streams vs strings in python3 is that it adds a strict type checking at runtime which isn't checked at 'authoring time'. Some APIs even make it impossible to do upfront type checking even if type hints would be provided (e.g. reading file content either returns a byte stream, or a string, based on the content of a string parameter in the file open function).

Recommended reading: [http://utf8everywhere.org/](http://utf8everywhere.org/)

<comment><USER29> > IMHO the whole python3 string mess could have been prevented if they had > chosen UTF-8 as the only string encoding instead of adding a strict string > type with a lot of under-the-hood magic.

That is basically what Python2 does, and it is completely wrong.

<comment><USER28> Can you give any reasons why this is completely wrong? The web seems to work just fine with UTF-8. The advantage is that you can pass string data around as generic byte streams without even knowing about the encoding. You'll only have to care about the encoding at the end points.

<comment><USER30> You are joking, right? Have you ever seen non-English webpages? More often than not, a multitude of ??? and Chinese characters pop up at some point or another.

<comment><USER28> I'm from Germany so I've seen a few non-English webpages. I can't remember having seen any text rendering problems since the late 90's or so.

<comment><USER31> this is because browsers have very sophisticated algorithms to detect the encoding because this was such a frequent issue. (and yes, UTF-8 adoption/support has been growing, which also helps)

being German and working in a multi-national company, i can confirm it is still very much an issue with software that doesn't handle this. Excel is one of the worst offenders, document corruption is rife especially when going between Excel on Windows and Excel for Mac. this is because Excel doesn't to UTF-8 as default for legacy reasons (I think), but also either doesn't have encoding detection or has very bad encoding detection.

</comment></comment></comment></comment></comment></comment><comment><USER32> > And the environment? [it's not even clear.] > [https://stackoverflow.com/questions/44479826/how-do-you- > set-...](https://stackoverflow.com/questions/44479826/how-do-you-set-a- > string-of-bytes-from-an-environment-variable-in-python)

That question is about interpreting backslash escape sequences for bytes in an environment variable. All this person wants is `os.environb` (and look, its existence highlighted a Windows incompatibility, saving them from subtle bugs like every other Python 3 improvement). [https://docs.python.org/3/library/os.html#os.environb](https://docs.python.org/3/library/os.html#os.environb)

<comment><USER33> Thanks, never noticed environb. I'm still learning new things about Python 3 ten years later.

</comment></comment><comment><USER34> Getting Unicode right, especially with various file systems and cross-platform implementations is hard, for sure. But, I think this quote:

"And, whatever you do, don't accidentally write if filetype == "file" — that will silently always evaluate to False, because "file" tests different than b"file". Not that I, uhm, wrote that and didn't notice it at first…"

shows a behavior that, to me, is inexcusable. The encoding of a string should never cause a comparison to fail when the two strings are equivalent _except for the encoding_. For example, in Delphi/FreePascal, if you compare an AnsiString or UTF-8-encoded string with a Unicode string that is equivalent, you get the correct answer: they are equal.

<comment><USER35> > The encoding of a string should never cause a comparison to fail when the > two strings are equivalent except for the encoding.

You'll have to admit that the encoding is a property of a string, just like the content itself. As always, you as a programmer are bound to know both of these properties to have predictable results. To compare two strings of different encoding to one another, you'll have to find a common ground for interpreting the data contained in the string.

If you don't want or need that, then all you have is a "string" of bytes.

<comment><USER34> Sure, but you can have defined rules about what happens when you compare values with disparate encodings, similarly to how you have to have rules about how column expressions are compared in SQL with regard to their collations. The way such things are done is typically to coerce the second value into the encoding of the first value, and _then_ compare the two values. What the Delphi compiler does is issue warnings when there might be data loss or other issues with such coercions so that the developer knows that it might not be safe and that they might want to be more explicit about how the comparison is coded.

</comment></comment></comment><comment> apk-d Let's be honest, the real mess is with UNIX filenames. I dare you to come up with a legitimate use case for allowing newlines and other control characters in a file name.

<comment><USER36> It's like a built-in unit test - devs have to _not_ mangle and assume anything about filenames they get from the system - though they still do, I've seen multiple times how my nice umlauts get mangled or my spaces cause scripts to fail.

<comment><USER37> A few years ago I tried naming my home directory with the unicode pile of poo () and a space in the name to test what of my code might break. However, it broke too much of third party tools/scripts that I occasionally needed for something, so I reverted within a few days.

Though it might be interesting to have an integration test box where the username (and thus all the relevant paths) includes all kinds of oddities - whitespace, emoji, right-to-left marker, etc.

</comment></comment></comment><comment><USER38> Going of on a tangent a bit here, but I think there are 2 important related issues:

* API design should fit the language. In a "high on correctness" language like Haskell or Rust, I'd expect APIs to force the programmer to deal with errors, and make them hard to ignore. In a dynamically typed language like Python where many APIs are very relaxed / robust in terms of dealing with multiple data types (being able to see numbers/strings/objects generically is part of the point of the language), being super strict about string encoding sounds extra painful compared to a statically typed language. I'd expect an API in this language to err on the side of "automatically doing a useful/predictable thing" when it encounters data is only slightly incorrect, as opposed to raising errors, which makes for very brittle code. Most Python code is the opposite of brittle, in the sense that you can take more liberties with data types before it breaks than in statically typed languages. Note that I am not advocating incorrect APIs, or APIs that silently ignore errors, just that the design should fit the language philosophy as best as possible.

* Where in a program/service should bytes be converted to text? Clearly they always come in as bytes (network, files..), and when the user sees them rendered (as fonts), those bytes have been interpreted using a particular encoding. The question where in the program should this happen? You can do this as early as possible, or as late as possible. Doing it as early as possible increase the code surface where you have to deal with conversions, and thus possible errors and code complexity, so that doesn't seem so great to me personally, but I understand there are downsides to most of your program dealing with a "bag of bytes" approach too.

<comment> dan-robertson I don't think Haskell is a very good example to promote for string handling. Things are mostly strict and well behaved once they make it into the Haskell program but before then they either need to satisfy the program's assumptions before being input or the program will be buggy/crash unless it is carefully written such that it's assumptions are right.

<comment><USER38> I didn't mention Haskell specifically for strings, but as a language that tends to be very precise about corner cases. That may not even be the best example, but I couldn't think of any better mainstream-ish language examples :)

</comment></comment></comment><comment><USER39> Indeed py3 decided to make unicode strings the default. This fixes all sorts of thorny issues across many use cases. But it does indeed break filenames. I haven't dealt with this issue myself, but the way python was supposed (?) to have "solved" this is with surrogate escapes. There's a neat piece on the tradeoffs of the approach here: [https://thoughtstreams.io/ncoghlan_dev/missing-pieces-in- pyt...](https://thoughtstreams.io/ncoghlan_dev/missing-pieces-in- python-3-unicode/)

Maybe handling the surrogates better would allow you to use 'str' everywhere instead of bytes?

</comment><comment><USER40> > For a Python program to properly support all valid Unix filenames, it must > use "bytes" instead of strings, which has all sorts of annoying > implications.

While in python 2, you had to use unicode strings for all sorts of actual text, which caused its own problems.

> What's the chances that all Python programs do this correctly? Yeah. Not > high, I bet.

Exactly.

</comment><comment><USER41> Don't think of python Unicode as a "string". Think of it as "text". I don't really understand the issues the author is having with things like sys.stdout and such because he did not provide complete examples. He should cite actual examples and bug reports that he has posted for these things, ive had no such issues. There's a lot of things we need to do to accommodate for non-ascii text but they are all "right" as far as I've observed.

</comment><comment> dan-robertson Part of the issue is to do with bytes and strings being considered totally different by python but confusingly similar to people.

The error from "file" != b"file" is particularly bad. It makes sense if you realise that a == b means a,b have the same type and their values are equal. But there is no way a even a reasonably careful programmer could spot this without super careful testing (and who's to say they would remember to test b"file" and not "file"). Other ways this could be solved are:

1\. String == bytes is true iff converting the string to bytes gives equality (but then can == become non transitive)

2\. String == bytes raises (and so does string == string if encodings are different)

3\. Type-specific equality operators like in lisp. But these are ugly and verbose which would discourage their use and so one would not think to use bytesEqual instead of ==

4\. A stricter/looser notion of equality that behaves as one of the above called eg === but this is also not great

<comment><USER42> > The error from "file" != b"file" is particularly bad. It makes sense if you > realise that a == b means a,b have the same type and their values are equal. > But there is no way a even a reasonably careful programmer could spot this > without super careful testing (and who's to say they would remember to test > b"file" and not "file").

I'm of the opposite opinion. I appreciate that b'a' != 'a'.

<comment> dan-robertson I don't think it's a problem that they aren't equal. This is reasonable. The problem is that it is hard for one to foresee this error. The mental model of bytes and strings is likely to be either their equal-looking literals or a mental concept of "bytes and strings are basically the same except for some exceptions." One cannot reasonably trace every variable to figure out whether it is a bytes or a string. The comparison a == b being false comparing strings to bytes makes sense when a and b could be anything. However when b is already definitely a (byte) string, it is more useful to get an error when a has a different type.

What is your opinion on numbers:

Should 1 == 1.0?

What about 1+0j?

Or 1/1 (the rational number, although I'm not sure this can be constructed)?

<comment><USER42> int and float being comparable is practical, though occasionally troublesome. Complex usually doesn't compare well across types. You can use a Fraction type for 1/1\. I haven't formed an opinion about that, since I don't use them often.

</comment></comment></comment></comment><comment><USER43> i love unicode handling in python3, it's so much better to work with. python2 was a mess, migrating old code requires looking at old code, the result is only better code, never a mess.

</comment><comment><USER44> > Python's unicode is a "mess" because of this single edge case I've > encountered

FTFY

<comment><USER45> more like

> Unicode is a "mess" because it can not unquote arbitrary backslash strings.

</comment></comment><comment><USER46> The article is about a specific instance (filenames). In general, handling Unicode as a bunch of indexable code points as per Py3 turned out to be not that great. I guess the idea came from the era where people still thought that strings could be in some sense fixed length. These days we better understand that strings are inherently variable length. So there is no longer any reason to not just leave everything encoded as UTF-8 and convert to other forms as and if required. Strings are just a bunch of bytes again.

<comment><USER6> The most correct way to expose Unicode to a programmer in a high-level language is to make grapheme clusters the fundamental unit, as they correspond to what people think of as "characters". Failing that, exposing strings as sequences of code points is a second-best choice.

UTF-8 is a non-starter because it encourages people to go back to pretending "byte == character" and writing code that will fall apart the instant someone uses any code point > 007F. Or they'll pat themselves on the back for being clever and knowing that "really" UTF-8 means "rune == code point == character", and also write code that blows up, just in a different set of cases.

And yes, high-level languages should have string types rather than "here's some bytes, you deal with it". Far too many real-world uses for textual data require the ability to do things like length checks, indexing and so on, and it doesn't matter how many times you insist that this is completely wrong and should be forbidden to everyone everywhere; the use cases will still be there.

<comment><USER47> That's silly. How often have you had to work with grapheme clusters without also using a text rendering engine? But the number of times you need to know the number of bytes a string takes, even when using scripting languages, is much higher. The only way to deal with this is to not make assumptions, and not have a string.size() function, but specific accessors for 'size in bytes', 'number of code points' and (potentially, if the overhead is warranted) 'nr of grapheme clusters'.

The 'fundamental' problem here is that the average programmer doesn't understand 'strings' because it seems so easy but it's actually very hard (well, not even hard, just big and tedious). Even more so now that many people can have careers without really knowing about what a 'byte' is or how it relates to their code.

<comment><USER32> > How often have you had to work with grapheme clusters without also using a > text rendering engine?

All the time. Want to truncate user text? You need grapheme clusters. Reverse text? Grapheme clusters. Get what the user thinks of as the length of text? Grapheme clusters. Not saying it's a good idea to make them any sort of default because of that, though; you're right that it should be explicit.

<comment><USER47> Truncating text is almost always (in my experience) a UI thing, where you pass a flag to some UI widget saying 'truncate this and that on overflow' and while rendering, it can then truncate using grapheme clusters.

How often does one reverse text? And when do users care about text length? Almost always (again, in my experience) in the context of rendering - <USER2> deciding on line length or break points, so when you know and care about much more than just 'the string' \- but also font, size, things you only care about in the context of displaying. Not something that should be part of the 'core' interface of a programming language.

I mean I think we agree here; my point was that I too used to think that grapheme clusters mattered, but when I started counting in my code, it turned out they didn't. Sure I can think of cases where it would theoretically would matter, but I'm talking about what do you actually use, not what do you think you will use.

<comment><USER32> I'm biased towards websites, but truncating text server-side to provide an excerpt is something I need to do pretty often. Providing a count of remaining characters is maybe less common, but Twitter, Mastodon, etc. need to do it, and people expect emoji (for example) to count as one.

Plus sometimes you're the one building the text widget with the truncation option.

<comment><USER48> Twitter's count of "characters" is code points after normalization[1].

I don't know who expects emoji to count as one character, but they'd be surprised by Twitter's behavior: something like [2] counts as 4 characters (woman, dark skin tone, zero width joiner, school).

[1] [https://developer.twitter.com/en/docs/basics/counting- charac...](https://developer.twitter.com/en/docs/basics/counting- characters.html) [2] [https://emojipedia.org/female-teacher- type-6/](https://emojipedia.org/female-teacher-type-6/)

</comment></comment></comment></comment></comment></comment></comment><comment><USER49> There are lots of comments indicating that the programmer is doing things wrong. But what is the right way to deal with encoding issues? Wait for code to break in production?

Whatever "best practices" there are for dealing with unexpected text encoding in Python, they do not seem to be widely well-known. I bet a large % of Python programmers (myself included) made the exact same errors the author had, with little insight as to how avoid them in the future.

</comment><comment><USER50> His examples are all stuff that isn't Unicode. The filename thing would probably work using a latin1 encoding, since that leaves 8 bit bytes undisturbed.

</comment><comment><USER51> That's not Python's fault - those are programmer errors.

Having said that, Python really has something to answer for with "encode" versus "decode" \- WTF? Which is which? Which direction am I converting? I still have to look that up every single time I need to convert.

Why the heck are there not "thistothat" and "thattothis" functions in Python that are explicit about what they do?

<comment><USER25> This is something I see folks trip on. I think encode/decode are fine names actually. The problem it's that Unicode strings have decode defined at all, and similarly, that byte strings have encode defined. Byte strings should only have a decode operation and Unicode strings should only have an encode operation. Depending on your input, the wrong operations can actually succeed!

<comment><USER31> Not sure what you're talking about mate, on Python 3.6:

        >>> "hello world".decode("utf-8")
        Traceback (most recent call last):
          File "<stdin>", line 1, in <module>
        AttributeError: 'str' object has no attribute 'decode'
        >>> b"hello world".encode("utf-8")
        Traceback (most recent call last):
          File "<stdin>", line 1, in <module>
        AttributeError: 'bytes' object has no attribute 'encode'

<comment><USER25> That's good. I guess it's only in Python 2 then.

<comment><USER52> In Python 2 it works almost the same, except you only get an error when encoding/decoding doesn't work out. So I see this as an improvement.

</comment></comment></comment></comment></comment><comment><USER53> If you're storing files with non-Unicode-compatible names, you should really stop. Even if on Unix, you _can_ technically use any kind of binary mess as a name, doesn't mean you _should_. And this applies to all kinds of data. All current operating systems support (and default to) Unicode, so handling anything else is a job for a compatibility layer, not your application.

If you write new code to be compatible with that one Windows ME machine set to that one weird IBM encoding sitting in the back of the server room, you're just expanding your technical debt. Instead, write good, modern code, then write a bridge to translate to and from whatever garbage that one COBOL program spits out. That way, when you finally replace it, you can just throw away that compatibility layer and be left with a nice, modern program.

In EE terms, think of it like an opto-isolator. You _could_ use a voltage divider and a zenner diode, but thats just asking for trouble.

</comment><comment><USER54> I can't believe there are still people whining about this in 2018.

Those problems with gpodder, pexecpt, etc. aren't due to Python 3, they're due to the software being broken. Without knowing the encoding, UNIX paths can't be converted to strings. It's unfortunate, but that's the way it is, and it's not Python's fault.

</comment><comment><USER55> The author has files with invalid names and complains that Python refuses to accept them. Maybe he should fix the names first?

<comment><USER56> If all tools he had access to behaved in the same way, he wouldn't be able to fix these "wrong" file names.

</comment></comment><comment><USER57> [https://web.archive.org/web/20181006121702/http://changelog....](https://web.archive.org/web/20181006121702/http://changelog.complete.org/archives/9938-the- python-unicode-mess)

</comment><comment><USER52> Author doesn't seem to care that there is a difference between Unicode the standard and utf-8 the encoding. While the changes on the fringes to the system are debatable, they are also in a way sensible. Internal to your application everything should be encoding independent (unicode objects in Py2, strings in Py3) while when talking to stuff outside your program (be it network, local file content or filesystem names) it has to be encoded somehow. The distinction between encoding independent storage and raw byte-streams forces you to do just that!

Stop worrying and go with the flow. Just do it as it is supposed to be done and you'll be happy.

</comment><comment><USER35> The encoding is a property of the string, just like the content, just as with any other object. If you want to compare strings with different encodings, you'll have to convert at least one of them.

I was never forced into encoding hell again, after reading this excellent post: [https://www.joelonsoftware.com/2003/10/08/the-absolute- minim...](https://www.joelonsoftware.com/2003/10/08/the-absolute-minimum- every-software-developer-absolutely-positively-must-know-about-unicode-and- character-sets-no-excuses/)

</comment><comment><USER58> I invest some karma to point out how I'd love for str to just use UTF-8 by default, and print as UTF-8 by default:

print(b'DONT b"EVERYTHING!"')

print(str(b'SAME!'))

print(str(b'I DONT WANT TO add ,"UTF-8" everywhere!','UTF-8'))

line="ום עולם"

output.write(line) # TypeError: a bytes-like object is required, not 'str'

fp.write(output.getvalue()) # TypeError: write() argument must be str, not bytes

Please at least allow us to set a global option via sys.setdefaultencoding('UTF8') as before to automatically encode/decode as UTF-8 by default!

</comment><comment><USER59> Dealing with string encoding has always been the bane of my existence in Python...going back over 10 years when I first started using it. I've never had such wild issues with decoding/encoding in other languages...that may be my privilege, though, since I was dealing with internal systems before Python, and then I got into web scraping.

Regardless, string encoding/decoding in Python is _hard_ , and it doesn't feel like it needs to be.

</comment><comment><USER60> I agree Python3 is an awful mistake and that straight-up Unicode is not well suited for storing arbitrary byte strings from old disk images. However, Python 3.1+ encode disk names as WTF-8 (aka utf-8b): [https://www.python.org/dev/peps/pep-0383/](https://www.python.org/dev/peps/pep-0383/) .

</comment><comment><USER61> This post barely scratches the tip of the iceberg.

For a more comprehensive discussion of unicode issues and how to solve them in Python, "Let's talk about usernames" does this issue more justice than I could write in a comment: [https://news.ycombinator.com/item?id=16356397](https://news.ycombinator.com/item?id=16356397)

</comment><comment><USER62> TFA is short and to the point. A few examples, a few links to other examples. Py3's insistence on shoving Unicode into every API it possibly could maybe fit, is often inconvenient for coders and for users. This thread has 100 comments, mostly disagreeing in the same fingers-in-ears-I-can't-hear-you fashion. Whom are we struggling to convince, here?

</comment><comment><USER63> If python programmers think they are the only ones with UTF problems, try Lazarus and Freepascal development mailing lists. The debates have going since forever, and I am sure issues will be popping up every now and then.

Try Elixir. According to their docs they've had it right from the word go - <USER3> think.

</comment><comment><USER51> Is the author saying that the Python programming language handles this badly, and all other (relevant) programming languages do not?

Or is that that Python's attention to detail means that issues that would be glossed over or hidden using ther languages are brought to the fore and require addressing?

</comment><comment><USER64> I just came from pycon India 2018. This is exactly what the keynote was about.(it was by author of Flask)

</comment><comment><USER65> Filenames are a good example to show people why forcing an encoding onto all strings simply doesn't work. The usual reaction from people is to ignore that and they'll shout: "fix your filenames!"

Here is another example: Substrings of unicodestrings. Just split a unicodestring into chunks of 1024 bytes. Forcing an encoding here and allowing automatic conversions will be a mess. People will shout: "Your're splitting your Strings wrong!"

The first language I knew that fell for encoding aware strings was Delphi - people there called it "Frankenstrings" and meanwhile that language is pretty dead.

As a professional who has to handle a lot of different scenarios (barcodes, Edifact, Filenames, String-buffers, ...) - in the end you'll have to write all code using byte-strings. Then you'll have to write a lot of GUI-Libraries to be able to work with byte-strings... and in the end you'll be at the point where the old Python was... (In fact you'll never reach that point because just going elsewhere will be a lot easier)

<comment><USER9> Sub strings of Unicode strings are fine. Byte level chunking of a Unicode string requires encoding this string as bytes, then working with bytes, then deciding the text.

Splitting a piece of Unicode text every 1024 bytes is like splitting an ascii string every 37 bits. It doesn't make sense.

</comment></comment><comment><USER66> It's just a not very well explained rant of some shitty libraries and a lot of legacy code. If you want to read about REAL complaints, read Armin Ronacher thought about it instead: [http://lucumr.pocoo.org/2014/5/12/everything-about- unicode/](http://lucumr.pocoo.org/2014/5/12/everything-about-unicode/)

</comment><comment><USER67> This is mostly why PHP6 wasn't a thing.

</comment><comment><USER68> Looks like we need a Python 4. (/s)

<comment><USER69> Well, Python already has a plan for Python 4. The Python 4 will be released after Python 3.8. There are already discussion on Python 4.0 in the dev group. It is just a new number after 3.8 so there won't be breaking issues like 2=>3.

<comment><USER70> I think that was just some idea that was discarded.

"Seems that we've reached the consensus: we release Python 3.10 after Python 3.9. We maybe release Python 4.0 at some point if there's a significant backwards incompatible change." [https://mail.python.org/pipermail/python- committers/2018-Sep...](https://mail.python.org/pipermail/python- committers/2018-September/006159.html)

</comment></comment></comment><comment><USER71> That sounds more like a mess handling things that are _not_ Unicode.

<comment><USER5> Yes, but the issue here would be that Python forced these "things which are not unicode" into unicode.

</comment></comment><comment><USER72> This is just the cost of using a dynamic language with implicit error handling (exceptions).

<comment><USER6> You can garble filenames just as easily in statically-typed languages. Consider, for example, Windows' infamous 16-bit units that aren't actually well-formed UTF-16. I'm not aware of any widely-used programming language whose type system will save you from that sort of thing ("here's some bytes, figure out if they're a string and if so what encoding" is a historically very difficult problem).

<comment><USER5> > You can garble filenames just as easily in statically-typed languages.

If the language assumes filenames are regular language strings, which not all do.

> Consider, for example, Windows' infamous 16-bit units that aren't actually > well-formed UTF-16.

unix filenames are literally just bags of bytes with no known or specified encoding.

<comment><USER6> Unix filenames don't pretend to be something they aren't. Windows filenames like to present a convincing façade of being UTF-16 right up until they aren't.

<comment><USER5> Windows filenames "like to present a convincing façade of being UTF-16" in the exact same way unix filenames "like to present a convincing façade of being UTF-8". Both are common assumptions neither is actually true, and all of that is well-documented.

<comment><USER73> > unix filenames "like to present a convincing façade of being UTF-8"

Except they never have? Unix paths have always been bags of bytes, both before Unicode and UTF-8 were invented and after. It's just convention that modern Unix systems use UTF-8 as the text encoding for paths.

<comment><USER5> > Except they never have?

And neither have Windows paths ever actually pretended to be UTF-16, that's my point.</comment></comment></comment></comment></comment></comment></comment>


Please Review my app - Css-Sprit.es - <AUTHOR>

Hello HN,<p>I'd really love to have your feedback on this little tool I made. It can help you producing your CSS sprites for all your web developements needs. <comment><USER2> Please add a "load image from URL" feature; don't make me upload my personal photos.

I uploaded a photo (bugs.jpg) and it showed up at the bottom of the screen, below the footer and outside the container! Firfox 3.5.2.

I am not exactly sure what it does; the only text that I thought would explain what it does only told me the generated code would be compatible with most browsers.

P.S. I like the giant buttons with the nice hover effect in the bottom; very tasteful.</comment>


HTML 5.2 is done, HTML 5.3 is coming - <AUTHOR> https://www.w3.org/blog/2017/12/html-5-2-is-done-html-5-3-is-coming/ <comment><USER2> Don't be fooled. The W3C does not even set the standard for HTML5 - it's developed by WHATWG and then periodically the W3C just makes a copy and throws away all the GIT history and erases the acknowledgments section and publishes it to keep themselves feeling relevant.

[https://html.spec.whatwg.org/multipage/](https://html.spec.whatwg.org/multipage/)

[https://stackoverflow.com/questions/6825713/html5-w3c-vs- wha...](https://stackoverflow.com/questions/6825713/html5-w3c-vs-whatwg-which- gives-the-most-authoritative-spec)

<comment><USER3> Oh man, how is this possibly legal? Can the WHATWG license their work in such a way as to disallow this, if you want a fixed version number to test against (which doesn't actually make sense, as there is no reference implementation, just various browser vendors that hopefully converge on and agreed implementation) you can just fork your own version of the WHATWG html spec, which will be more inline with browsers and have further bug fixes and corrections.

<comment><USER4> Why? WHATWG doesn't make money with the spec itself, rather want it spread to wider audience, so more are informed of the upcoming changes for smoother deployment.

<comment><USER5> Because that's just how copyright works. You can't erase attributions and redistribute without permission.

</comment></comment></comment></comment><comment><USER6> I already suggested basing the W3C spec on the web developer edition of WHATWG HTML as a compromise. In retrospect, the full spec was never a good fit for a W3C REC in the first place. (you may remember the 2022 prediction)

</comment><comment><USER7> I thought we dropped the "5" and continued with just "HTML" as a living standard?

<comment><USER6> That is WHATWG not W3C. I don't like versioning HTML either, to be honest.

</comment></comment><comment><USER8> so when do we expect to see mass browser adoption of this?

<comment><USER9> That's the {amazing|horrifying} nature of this: Web Standards are all descriptive. The standard is only written _after_ the majority of browsers already shipped it, and enough major sites rely on it.

(Of course, this means by the point that you actually get to have a discussion about standards, you can't really change anything more, and all standards committees, from WHATWG to W3C, are just rubberstamping whatever the browser vendors want)

<comment><USER10> Here is my reply the last time this concern came up: [https://news.ycombinator.com/item?id=15927069](https://news.ycombinator.com/item?id=15927069)

<comment><USER9> I can recommend to read this comment chain as well, IMO it shows the arguments of both sides very well (and avoids us having to re-argue this every time).</comment></comment></comment></comment>


Hidden Government Scanners Will Know Everything About You From 164 Feet Away - <AUTHOR> http://gizmodo.com/5923980/the-secret-government-laser-that-instantly-knows-everything-about-you

<comment><USER2> Is this a joke? It _looks_ real - but can anyone comment on real life applications of Coherent anti-Stokes-Raman spectroscopy (and other techniques listed below)

Yes, I can beleive it works in the lab, on static samples,. But a dust fragment on a lapel as someone walks through the airport?

I mean, otherwise, they just invented the Star Trek Tricorder.

The company seems real and serious: [http://www.geniaphotonics.com/business- markets/defense-and-s...](http://www.geniaphotonics.com/business- markets/defense-and-security/ds-chemical-detection-sensing/)

Edit:

I think I would like to see the following law: Any scanning, recording, tracking (etc) done in a public location or place of work _must_ make the results public and interrogatable in (real/reasonable) time.

We will never stop these technologies, and frankly some of them seem massively beneficial (this could test blood samples on your doctors desk). But we should not allow the results to be hidden, kept by the police or our employers.

A general law of freedom, much like to general law of net neutralitry we want as well.</comment>


Ask HN: Should I go back and finish my CS degree? - <AUTHOR>

I attended university for a degree in CS right out of high school but never finished, completing about the first 2 years. After dropping out and wandering around for a bit I eventually went and did a 3 year college Systems Analyst program. I graduated with great grades and have since had a couple jobs working as a software developer and am currently working for a large financial institution. I am fairly happy with my job but I feel like I am kind of stuck doing the same web application/SOA web service type work. I would really like to work on some more interesting problems where I get to actually use some cool algorithms. I have been considering some options for completing my bachelors degree in CS, but I am just not sure if it is something that I really need or not in order to move to where I want to be in my career. I spent a lot of time on my while at college sort of trying to play catch-up on the stuff that I missed not finishing my CS degree. I feel I have a fairly strong background in CS despite not finishing; I have written neural networks and parser generators for fun in my spare time and have a fairly good understanding of a lot of the basic CS concepts such as data structures/algorithms, time complexity, graph theory etc. I currently do a lot of web related stuff but I have done a fair amount of C/C++ as well. If anything I feel my weakest point is math. I know basic calculus and linear algebra but nothing advanced really. My strongest skill is probably software engineering in general (design patterns and whatnot). Is there more that I would have learned from finishing my degree that is essential? Do you really need that piece of paper, or can just having enough experience and the knowledge itself get you the job of your dreams? <comment><USER2> _I would really like to work on some more interesting problems where I get to actually use some cool algorithms_

You could try working at a startup, or starting your own. A big financial institution is able to hire people for specific roles. If they need someone to focus on algorithms, they will hire a top CS grad. If they need someone to do web design, they hire you. At a startup, you get (have) to do a little of everything, so you'll get more experience there.

 _Do you really need that piece of paper, or can just having enough experience and the knowledge itself get you the job of your dreams?_

No, you don't need the degree. With a BA in English, I am on about the same footing as college dropouts in the programming world, but my experience speaks for itself. I've worked with many CS majors who learned how to talk about algorithms and data structures, but I have demonstrated that I can ship code actually using them.

</comment><comment><USER3> My vote is yes. I went to UW Comp Sci department and it's been a great learning experience that gave me the ability to understand the entire stack - from hardware up.</comment>


WordPress on .NET - <AUTHOR> http://www.peachpie.io/2017/02/wordpress-announcement.html <comment><USER2> This seems to be built on an open-souce PHP->CLR compiler called Peachpie[1]. Interesting project. I don't have any PHP codebase that I still maintain but looking forward to trying it with some scripts I've written back in the days. I wonder if it would be faster than PHP 7 - which is already very fast (ok, just found these: [http://www.peachpie.io/benchmarks](http://www.peachpie.io/benchmarks) ).

[1]: [http://www.peachpie.io/](http://www.peachpie.io/)

[edit] If the author is looking, a few nitpicks from the website:

\- A blank page is shown if browser doesn't load 3rd party scripts by default

\- After enabling them, the scroll wheel doesn't work

\- Typo: "free completely to use" -> "completely free to use"

\- "Run PHP apps on the most secure platform available, Microsoft .NET" -> I also admire the platform but most secure? Comes off a bit too strong.

<comment><USER3> >I also admire the platform but most secure? Comes off a bit too strong.

There aren't a lot of .NET exploits for a variety of reasons. Its pretty much the Java of the MS world so a lot of issues more liberal languages have, it doesn't, especially in regards to buffer overflows. Its attack surface is also pretty low as it only really inter-operates, by default, with a limited number of MS applications and is frequently the most common setup, while PHP supports anything and everything. Http.sys also seems like a more secure product than openssl.

That said, its also the minority platform so it benefits from a lack of effort to target it. I also worry that the closed nature of the platform could lead to someone(s) building up an army of zero days while the more open stuff gets discovered and disclosed faster. I'm not sure how realistic that fear is.

At my job we're always scrambling with our FOSS architecture in regards to emergency patching. The MS stuff is more or less 'do monthly patches and forget.'

<comment><USER4> > Http.sys also seems like a more secure product than openssl

How is a _goddamn HTTP parser in THE KERNEL_ more secure than a crypto/TLS library?

http.sys is a fundamentally horrendous idea, see CVE-2015-1635 [https://technet.microsoft.com/library/security/MS15-034](https://technet.microsoft.com/library/security/MS15-034) An exploited HTTP parser shouldn't lead to RCE "in the context of the System account".

<comment><USER5> The .NET Core example in the article uses the ASP.NET Core Kestrel server which operates entirely in user mode.

<comment><USER4> I know about Kestrel, I'm responding to that particular phrase in the comment :)

</comment></comment></comment></comment></comment><comment><USER6> I've been following the peachpie project for a while. The effort they put in is very impressive.

They choose to heavily modify the C#/VB compiler (Roslyn) to handle php syntax. Microsoft should be investing time into helping them succeed, it would be great to see the Roslyn compiler platform become more broadly used.

<comment><AUTHOR> Microsoft is doing quite a bit of supporting, can't complain there :)

</comment></comment><comment><USER7> I work at a large company and we are starting to branch out into other languages. the problem is most of our infrastructure in .NET. Something like this would be really cool as it would allow us to utilize PHP while not asking too much of infrastructure team. I might put in an OSS approval request for this.

<comment><USER8> I think you might be the first case I've heard of of someone wanting to go from .NET to PHP.

<comment><USER9> Most of my work in the past few years has been migrating .NET projects to PHP for enterprise clients. Especially with PHP 7, the performance is good, it's easier to hire for, and the total cost of ownership is lower. The doomsayers predicting poor security are wrong - it is just as possible to build a secure PHP system as a secure .NET system, and just as easy to build an insecure one.

Of course, it depends on the project and the company, so this isn't a universal truth. But there are definitely a lot of companies out there spending a fortune on a .NET stack that are looking to move to a PHP open source stack for good reasons.

<comment><USER10> Not sure that's true. Out of the box ASP.NET has request validation enabled by default that screens out lots of opportunities for XSS [https://www.owasp.org/index.php/ASP.NET_Request_Validation](https://www.owasp.org/index.php/ASP.NET_Request_Validation) PHP doesn't have that for example.

<comment><USER11> > PHP doesn't have that for example.

Bare PHP, sure, but pick basically any framework that any modern PHP-based web app is built on and then it does, no?

<comment><USER8> Symfony certainly offers it.

</comment></comment></comment></comment></comment></comment><comment><USER12> Running PHP and indeed WordPress on the .NET CLR seems like a great way to eliminate a whole tranche of bugs/vulnerabilities in one swoop. Respect.

<comment><USER13> How so? A PHP runtime running WP on .NET needs to be bug-for-bug compatible with the real thing (and I've heard early PHP APIs aren't exactly a paragon of API design).

Also, why would php-on-net be updated more often than one of the most used runtimes on the web?

</comment></comment><comment><USER14> This is amazing and from the comments I've heard that Microsoft is around supporting the development of this? Is there any MS material mentioning this effort? I'm really impressed by your work

<comment><USER15> there are links to MS blogs and livecasts [http://www.peachpie.io/resources](http://www.peachpie.io/resources) and of course youtube channel [https://www.youtube.com/channel/UC5Hh61n9HFyqYbU4iFkqy4A](https://www.youtube.com/channel/UC5Hh61n9HFyqYbU4iFkqy4A)

</comment></comment><comment><USER16> <3 this! any idea on when it will be production-ready?

<comment><USER17> What's the benefit over running a standard PHP-Interpreter?

Haven't used PHP or WordPress since 2010 so I might have missed something but to me this looks more like a tech demo then something one would use in a production environment.

<comment><USER18> I remember from the PHP on Java that there were huge performance benefits. So large PHP hosts should see benefits.

<comment><USER19> PHP 7 improved performance by around 3x compared to PHP5, so I wonder how true that still is.

<comment><USER20> The benchmarks [http://www.peachpie.io/benchmarks](http://www.peachpie.io/benchmarks) show peachpie outperforming PHP7 easily by 100x.

<comment><USER9> On Azure, without Opcache enabled. That's not a very good comparison.

<comment><USER21> While I do know that Azure is owned by Microsoft I'm not so sure on why it's such a valid argument.

In the end it's not like they're actively trying to ruin languages other than C# on their servers? Especially since that would ruin it for a lot of customers?

<comment><USER9> I would guess that .NET is more optimised on Azure, whereas PHP is not. So you're comparing an optimised installation of .NET against an unoptimised installation of PHP. In general, a benchmark saying "compares various metrics" is pretty meaningless.

<comment><USER21> I know, but is there any proof of this? And how can we be sure that the .NET environment is better optimised than the PHP environment? I would find it weird for them to shoot themselves in the foot like that. Since if someone were to make a decent blog post that showed proof they'd only have C# programs on Azure...

This all being said they're pretty transparent with their benchmarks. They supply all the code that they've used and are even comparing it to another project which aims to do the same.

I guess I'm just a bit angry because people are just saying "It's unfair" while they can easily do the tests themselves and then call someone out for it.

<comment><USER9> It's less about being unfair, more about it lacking much value as a benchmark.

The context of the parent to which I originally replied was suggesting that .NET was 100x faster than PHP 7. So I'm pointing out that claim is rather like saying "I'm faster than Usain Bolt". You're right that I don't think the original article was making such a claim.

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER22> I don't care about performance, show me if security has improved!

(which in the case of Wordpress can't be that hard)

<comment><USER23> WordPress security is complex.

The core code is OK, security wise. Comparible to similar projects, I guess.

The two major issues are that typically PHP runs with permission to modify the WordPress directory, which is useful for automatic upgrades, but means any exploit in any plugin, theme, etc. instantly becomes, "can replace WordPress".

The other is that all themes and plugins are completely unsandboxed, and the quality is extremely variable.

The permissions thing can be fixed, by running a very stripped down wp-php user with only read access to the code, and only write access to wp- contents/uploads (and a logging dir). Then you do automatic upgrades with wp- cli and (real) cron from a user with write access.

The quality of plugins and themes is not easily fixed.

The API for writing plugins and themes doesn't help. It's archaic, spaghetti, and doesn't have any kind of coherenc. Global functions like wp_is_home(), the_loop(), get_sitename(), etc.

I've come across themes which bundle joomla inside them as they're really joomla themes with a shim.

<comment><USER24> > OK, security wise

Since when is implementing your own home-grown shitty replacement for parameterised queries "OK"?

<comment><USER25> Since you need to support versions of PHP that don't support the extensions that have parameterized queries.

<comment><USER4> Except… you don't?

> WordPress also works with PHP 5.2.4+ and MySQL 5.0+, but these versions have > reached official End Of Life and as such may expose your site to security > vulnerabilities

> PDO ships with PHP 5.1, and is available as a PECL extension for PHP 5.0

Unless you mean "hosts that disabled PDO"… I think they can safely be ignored.

<comment><USER25> PDO is disabled by default, and was even on default installs of 5.6 last I checked.

<comment><USER24> Most Linux/BSD distributions seem to separate out "core" PHP extensions (those that are distributed with php source, not via peel) into individual packages.

So while PDO may not be installed by a plain `apt-get install php5` or similar, I doubt the now deprecated `mysql` extension is installed by "default" in those scenarios either.

Edit: this approach also means that the PDO extension in php.ini _will_ be commented, because its loaded by a package specific ini file e.g. /etc/php/7.0/fpm/conf.d/* which are generally symlinks to /etc/php/7.0/mods- available/

<comment><USER4> Yeah — 'mysql' wouldn't be installed either on FreeBSD if you just install php. So typically hosting providers install all the extensions.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER26> This is very clever, what is the performance like compared to PHP?

<comment><USER27> From their benchmarks page (which looks remarkable)

[http://www.peachpie.io/benchmarks](http://www.peachpie.io/benchmarks)

<comment><USER28> Remarkable, but as bleeding-edge tech I think it's disingenuous to include PHP5.6, which is now only on security support. I know many PHP sites run this version and even older ones, but I think the audience for a project like this are PHP7 developers.

That said, I was really impressed by the improvement on require performance. The requests/sec also seem to have big gains.

I also wonder what security improvements other are expecting from this? Isn't unsecure code unsecure regardless of language?

<comment><USER27> CLR imposes lot of requirements for an assembly to run, that would definitely improve security esp. when comparing it to a interpreted language.

<comment><USER28> Ah ok. In terms of security I don't tend to think of memory safety, etc. especially since PHP processes die fast. I thought everyone was talking about SQL injections and the like so I was very confused

Thanks for clarifying!

<comment><USER27> [https://www.cvedetails.com/vulnerability- list/vendor_id-74/p...](https://www.cvedetails.com/vulnerability- list/vendor_id-74/product_id-128/PHP-PHP.html)

Wow those are interesting.

SQL Injection is still an issue if there are string concats every where and calling the DB, so that is something, that can still happen.

</comment></comment></comment></comment></comment></comment><comment><USER29> Any chance we can get PHP's HashDoS vulnerability removed if we switch to PeachPie?

PHP 7.2 will make libsodium a core extension, so if you use that, you can make use of SipHash-2-4.

<comment><USER15> Interesting question. Beside that ASP.NET limits request size, Peachpie uses .NET hash function that differs from that used in PHP. Am I close ?

</comment></comment><comment><USER30> Several years ago I used to run a WordPress blog on IIS 7 using FastCGI. The performance wasn't great but acceptable for a small blog. Wonder how this compares.

<comment><USER31> It is faster

</comment></comment><comment><USER32> Your developers were so preoccupied with whether or not they could, they didn't stop to think if they should.

<comment><USER33> That's an assumption on your part. What a terrible comment.

<comment><USER34> He's quoting Jurassic Park. The idea is that the technology does something that would normally be considered highly unusual or even ridiculous to consider, like running a very complicated code base through a transpiler to make it run on another tech stack. Generally speaking, this kind of comment would be a compliment. Sort of along the lines of "wow, that's absolutely crazy, you did something no one would have thought possible, good work".

It's a much more light-hearted comment than you seem to think it is.

<comment><USER33> I don't care what he's quoting and to see this as 'light-hearted' (without any further elaboration on the part of the OP) is extrapolation.

FWIW the comment in the movie is _also_ a negative one.

> John Hammond: I don't think you're giving us our due credit. Our scientists > have done things which nobody's ever done before...

> Dr. Ian Malcolm: Yeah, yeah, but your scientists were so preoccupied with > whether or not they could that they didn't stop to think if they should.

> John Hammond: Condors. Condors are on the verge of extinction...

> Dr. Ian Malcolm: [shaking his head] No...

> John Hammond: If I was to create a flock of condors on this island, you > wouldn't have anything to say.

> Dr. Ian Malcolm: No, hold on. This isn't some species that was obliterated > by deforestation, or the building of a dam. Dinosaurs had their shot, and > nature selected them for extinction.

> John Hammond: I simply don't understand this Luddite attitude, especially > from a scientist. I mean, how can we stand in the light of discovery, and > not act?

> Dr. Ian Malcolm: What's so great about discovery? It's a violent, > penetrative act that scars what it explores. What you call discovery, I call > the rape of the natural world.

<comment><USER32> You are far too serious a person.

I actually stand by the sentiment. Wordpress is a virus, and like jQuery, it has negatively impacted the development scene of the language is lives in. PHP, on the cusp of becoming a more respectable language (I freely admit my love for PHP, to this day, even though I primarily work in Java and Python now), was taken over by WP "developers" who flooded the market and sell themselves as engineers when all they can do is copypasta a few wordpress idioms.

I don't hate on wordpress itself, its a fine, though early on troubled codebase. The predatory community it has created, the security issues it continues to create, and the low-barrier to entry that floods the market with a ton of really bad code isn't something I would want to bring into another language's world.

And here's where the light-hearted part comes in... I know that projects like these would NEVER actually cause that kind of effect on their language because nobody would take it seriously... but the analogy of what _COULD_ happen (you know, fictional, like ridiculous unrealism of the base concept of the movie) was apt.

I fail to understand why in a world so caught up in negativity and rigidness, people chose to not allow themselves to enjoy or notice light-hearted humor, and actively defend their stance against it, as if somehow eliminating light hearted jest is a benefit to a world already buried in such negativity...

<comment><USER33> The person posting this went out of his way to try to do some good. Instead of taking them down with cheap shots: What have you done to give the PHP world (better?) options?

If everybody that railed on others creations would instead spend the time to provide better alternatives then we wouldn't be having this discussion. But cheap shots are cheap and work is hard.

</comment></comment></comment></comment></comment></comment><comment><USER35> Well, considering the fact that PHP post v4 is trying to become Java, it made sense that at some point it would be able to run on the JVM or its step-cousin CLR.

By the way, does anyone know of a similar project where the JVM is targeted instead?

<comment><USER36> Quercus was (is?) an attempt at doing PHP on the JVM, it's been many many years since I last looked into it. [http://quercus.caucho.com/](http://quercus.caucho.com/)

</comment></comment><comment><USER37> I can't file an issue on GitHub but the project is missing the license and the WordPress standard copy seems to have been removed.

<comment><AUTHOR> We disabled issues for now, but thanks for pointing this out. Just added the license to the peachpie-wordpress repository.

</comment></comment><comment><USER38> Cool idea. How is the performance vs PHP?

<comment><AUTHOR> Check out Ben Adams' benchmarks at [https://github.com/benaadams/PeachpieBenchmarks](https://github.com/benaadams/PeachpieBenchmarks)

</comment></comment><comment><USER39> But why?

</comment><comment><USER40> This is a great way for Microsoft to show the potential of and build awareness for .NET core, I hope they are helping these guys.

The nuget thing is inefficient, MS should plot a path to make npm a first class citizen in all ms tools.

<comment><USER40> >NPM is already supported by default for quite some time in VisualStudio

I should have been more clear. I mean MS should work with Node/LAMP people to invest in one package manager more than the other. Like they did with TypeScript, they didn't just build a tool that linux people never used, they made useful tooling and also spent a lot of time bringing Google and others into the fold.

It makes sense to back npm because (guessing) it's used more than nu-get is used by VS devs. However if nu-get has a stronger ecosystem, stronger user base, etc choose it.

Having competition is usually good, but sometimes it just means it's more work to have to learn more/work more to get the same thing done (if you work across environments). Take instant messaging protocols, lots of competition but not even having cooperation has made the result terrible. Incompatibility everywhere, reinventing the wheel, etc.

To be sure some things are better not unified. We have multiple, un-unified browser engines and implementations which is critical. It improves competition, innovation and quality while still allowing surface level standardization in html/css/etc.

I try to look at each set of competing projects independently and ask: Overall is this helping, or it is mostly just wasting time as clients are forced to learn multiple techs.

</comment></comment><comment><USER41> No.

</comment><comment><USER42> PHP is about to face Embrace, extend and extinguish?

<comment><USER4> Shared Apache+mod_php+MySQL hosting turns out to be impossible to extinguish…

<comment><USER13> Yes. I had hoped node.js would fill the role of a cheap generic hosting platform with taste. But while it works well for me on VMs/cloud, managed hosting for node.js is nowhere near PHP in terms of ubiquity.</comment></comment></comment>


Firefox and Thunderbird: A Fork in the Road - <AUTHOR> https://marksurman.commons.ca/2016/04/25/firefox-and-thunderbird-a-fork-in-the-road/ <comment><USER2> It's sad to see that Mozilla considers desktop email clients as outdated. It's true that most people use phone apps and webmail, but in enterprises the story is completely different. MS Outlook rules in many places, but I'd bet there are at least a few million people using Thunderbird as their desktop mail client, more so in a (slightly "hostile") MS Exchange environment.

If Thunderbird were to really evolve and provide what people have been asking for for a very, very, very long time, it is good calendaring that "just works" with MS Exchange (I know about the extensions and external applications available, but none of them provide a seamless experience or all features that are required for calendaring).

Post Edward Snowden's revelations, more people need a good way to do encrypted mails well too (that's complex and is also more of a user experience issue, but an organization like the erstwhile Mozilla Messaging could have spent a good amount of effort on it). Along with this, we would have encrypted chats (since chat is already part of Thunderbird), encrypted voice calls, etc. That would be phenomenal!

IMO, Mozilla really lost focus on Thunderbird long before it decided to shelve it as a "community driven project".

<comment><USER3> Mozilla of today is run by designers, webheads and "social" personas.

As for encrypted email, it could be made very simple. When you set up an account in Thunderbird, have it ask a simple question "do you have an encryption key?". If the user hits no, generate the key pair, and insert the public key by default.

Then starts harvesting public keys from incoming emails, and encrypt by default if sending to an address that has a known public key.

<comment><USER4> Doesn't this sort of abandon the trust model of PGP? If you make default encryption a part of the system and an elevator pitch bullet point, I don't think it's really fair to call it simple because a UI change would be needed to ensure that though email may be encrypted, users can't necessarily trust the identity of the sender.

<comment><USER5> In the status quo nothing is verified or encrypted anyway. When we make the perfect the enemy of the good, we reinforce that status quo. As long as the trust model of PGP depends on everyone to get the same religion, we are stuck waiting for a messiah which will never come. If we have any hope of improving the status quo, we need some kind of bootstrap, and that bootstrap won't look exactly like the final result we want.

Ubiquitous email encryption would improve on the status quo, and prepare the ground for further improvements. People who don't care would still be generating more verifiable information than the status quo. People who want to use encryption wouldn't be forced to drop it in order to talk to family and friends. People who want to carefully manage their identities and use signing to build a web of trust could build that on top of ubiquitous encryption. This would get us closer to the PGP trust model than anyone has been able to do purely by evangelizing the PGP trust model.

<comment><USER6> After so many years of email that's entirely in the free and clear, I'd consider _any_ steps nearer encryption as beneficial. Even if fundamentally weak improvements with annoucements littered with caveats for those in the know.

I'd like for encrypted email to become a standard thing in my lifetime. We are no nearer than when I trialled PGP in the 90s.

Then we can think about email that doesn't leak all your meta-data.

</comment></comment></comment></comment></comment><comment><USER7> A problem I foresee is Firefox's attempts to move away from XUL.[0] Firefox might just about manage that move, but who will do that work for Thunderbird? And once Firefox moves away from XUL, who will maintain XUL for Thunderbird's sake?

[0] Firefox is still XUL-based for now, but I think the writing's on the wall given the security issues with XUL and XPCOM, and Firefox's move away from XUL/XPCOM-based extensions.

<comment><USER8> Come over to Pale Moon [[https://www.palemoon.org/](https://www.palemoon.org/)] (firefox fork continuing XUL) and FossaMail [[https://www.fossamail.org/](https://www.fossamail.org/)] (thunderbird fork based on Pale Moon Goanna engine).

These products aren't perfect, but they allow the traditional strengths of firefox (powerful extensions based on XUL/XPCOM) as opposed to the new chromification of the firefox extension api. Also, you don't have the mozilla politics and constant moral hectoring (if you read the firefox start page or go to mozilla's website, I think you might know what I mean).

<comment><USER3> Frankly I am pondering going back to Seamonkey (what used to be the Netscape/Mozilla application suite).

<comment><USER9> SeaMonkey likely has even fewer resources to maintain the mail engine within than Thunderbird does.

</comment></comment></comment></comment><comment><USER10> I honestly hope that LibreOffice will take over the development of Thunderbird. They seem to be doing an excellent job at running a successful open source project and Mozilla looks like it lost its course a couple of years ago (before someone asks me why I think that: [https://blog.r3bl.me/en/mozilla- dissatisfaction/](https://blog.r3bl.me/en/mozilla-dissatisfaction/))

<comment><USER11> There are some parallels between LibreOffice and Thunderbird that make The Documentation Foundation's adoption of Thunderbird attractive.

1\. Both LibreOffice and Thunderbird are desktop implementations of applications for which most of the recent momentum has been behind web-based implementations (Google docs, GMail, Office 365, etc.). So TDF is used to the desktop vs. web dynamic.

2\. Both LibreOffice and Thunderbird has large installations in business/government environments.

3\. Both LibreOffice and Thunderbird started out being governed by different organizations and would be adopted by TDF (slightly different situation for LibreOffice which was a fork, though Thunderbird will be forking off its shared base with Firefox). So TDF is used to adopting legacy code and continuing to evolve it.

That said, I hope the TDF scrutinizes Thunderbird and only takes it on if it is confident that Thunderbird fits into its overall mission and infrastructure. In recent times I have liked what I have seen from TDF and been somewhat disappointed by Mozilla, so I wouldn't Thunderbird to hamper LibreOffice's momentum.

</comment></comment><comment><USER12> I get the sense that Thunderbird doesn't have cohesive vision or leadershihp: All the big organizational moves seem to be Mozilla's initiative; I don't hear about much happening at Thunderbird, technically or organizationally; and Simon Phipps' report[1] says the following when considering an independent Thunderbird Foundation, and the implication I take is that these issues can't be resolved now:

        *   Creating a new entity involves resolving all of the
            issues currently facing Thunderbird and its community
            and then making them concrete in legallybinding bylaws
            and articles of association.  As such, it is a step best
            taken after these things have been resolved, rather than
            before.

        *   With all these things taken into account, I would not
            recommend this option as a first step.  However, it may
            become appropriate in the future for Thunderbird to
            separate from its new host and become a full independent
            entity, and the ability to do this should be considered
            in selecting a new home.

But my concern is based on only an intution plus a few observataions; does anyone know? I don't mean it critically; it's hard to organize and find leadership resources for a volunteer project. Mostly, I'm just concerned about Thunderbird.

[1] [https://blog.mozilla.org/thunderbird/files/2016/04/Finding-a...](https://blog.mozilla.org/thunderbird/files/2016/04/Finding- a-Home-for-Thunderbird.pdf)

</comment><comment><USER13> Mozilla is a weird organization.

Thunderbird makes client-side PGP encryption a little easier at a time in the world where not only governments are harvesting every cat-gif email we send, but also those Valley companies who harvest and analyze practically everything we do, just to send us "better ads".

I do commend those volunteers who didn't sit on the internet and makes comments like the rest of us, but took charge and kept Thunderbird alive.

<comment><USER14> Mozilla lives on money from the ad industry, so in a way they are all part of the same system.

<comment><USER4> It seems like the whole modern connected infrastructure we rely on relies on money from the ad industry. Google, webmail services, free websites, all of them rely on a small subset of companies that buy eyeballs. From an outsider, it certainly seems like Mozilla is the only company in the space that's trying to make the Internet better, rather than making the browser a path into their own ecosystem of services (and consequently ads).

<comment><USER15> Firefox was created to be a browser in which the developers and the community can have a voice, rather than having all decisions made by the enterprise. I know there are many heroic stories and tales, but after all those are the two goals.

Why are there laundromat, 99 cent store, Deli place and a bus stop next to a street full of homes? Next to a college campus you'd find Deli and street food vendors? Of course one can give out free food and you can always choose to work from home so you never have to deal with pedestrian jam or the smoke coming from the grill. For example I can skip WSJ paywall by not registering any account and not reading any WSJ online articles. That's fine and the anology of this is finding a suburban home. The public Internet is like New York City, so many people, so many business, so much to navigate (but of course as a NY resident I can get tired of the city myself). If you want no ads, you have to lobby a group of politicians (actually companies) and convenience them your plan is better and is more profitable. The thing is that once there is a collation of business opportunity, uniting that knot is very difficult because everyone has a piece in it. Few people want to bet on new plan.

<comment><USER12> > Firefox was created to be a browser in which the developers and the > community can have a voice, rather than having all decisions made by the > enterprise. I know there are many heroic stories and tales, but after all > those are the two goals.

While I like these ideas, this isn't quite accurate. That may describe Mozilla Application Suite (now Seamonkey) to some extent, but not Firefox. Mozilla's mission is to promote the open Internet, as opposed to the proprietary one (originally represented by IE's market dominance):

[https://www.mozilla.org/en-US/mission/](https://www.mozilla.org/en- US/mission/)

</comment></comment></comment></comment></comment><comment><USER16> Firefox disrupted browser market and now Chrome/IE are making a trio not too far from each other, it is now a uphill battle for Mozilla.

Thunderbird is the only choice across platform for Desktop usage. I use it daily and don't mind paying for it.

On the other hand, since Microsoft is porting everything to Linux nowadays it may one day port Outlook to Linux, who knows though I have never trusted them and I have not used it for centuries.

Please keep Thunderbird evolving ahead.

<comment><USER3> Not likely. All MS have done towards Linux have been aimed at Linux on the server/cloud.

Their latest move is about retaining web devs that use Ubuntu LXD for their cloud servers by giving them a desktop development environment within Window that match their cloud environment.

Full blown Linux on desktop, and mobile (hello android), is still under attack.

</comment></comment><comment><USER17> I depend on Thunderbird and have for many years. It's a great mail client. I hope it continues to be a great mail client.

<comment><USER18> I do too and I don't know what I would replace it with if it becomes no longer viable.

What other "serious" email clients are out there for Mac?

<comment><USER19> Postbox is a pretty great Thunderbird derivative. [https://www.postbox- inc.com/](https://www.postbox-inc.com/)

I love Airmail, but require something cross-platform. Postbox isn't as pretty, but is cross platform, priced right, and has great default config out of the box.

<comment><USER9> I'd recommend against PostBox. It's a Thunderbird derivative, so it has the same security vulnerabilities as Thunderbird does. But it's not updated regularly. The current build of PostBox was last updated in November 2015. It appears to bundle Gecko (Mozilla rendering engine) 7.0.1 which was a part of Firefox/Thunderbird 7 back in 2011. And the only version of Lightning (calendar and tasks) for PostBox is "experimental and unsupported" and was last updated in 2012.

You're better served using Thunderbird and any extensions you'd like to tweak things a bit.

</comment></comment></comment></comment><comment><USER20> Keep hard charging on Firefox, Mozilla. If all you have resources and focus for is the browser, then so be it.

Firefox is your iOS -- if you need to gut the Lion team for a year or three in order to ship a stable world-beater, then DO THAT!

Maybe circle back to email in a few years after Firefox stabilizes, that would be cool too.

</comment><comment><USER21> "Success for Firefox means continued relevance in the mass consumer market as a way for people to access, shape and feel safe across many devices."

Gah! No! It should mean that Firefox continues to be an awesome, customizable web browser! Why does all this wishy-washy "feelings" crap have to creep into their mission statement?

<comment><USER22> "feel safe" is not "wishy-washy "feelings" crap". It has all sorts of implications - privacy, security...

<comment><USER23> Ya, well I'd rather be safe than feel safe.

</comment></comment></comment><comment> norea-armozel I'm not surprised Thunderbird is breaking off considering how much the Web- only approach to just about every service is in full effect. How do you serve up ads in an email client without pissing everyone off? Obviously you can't do it so Thunderbird has to go!

Also, for most people the email address they have isn't for legitimate correspondence considering how many people use Snap Chat, FB Messenger, WeChat, and the like to keep in contact with people. Why bother making an email client even it's available in the app store (phone) when folks will sooner install an IM app? It's not to say email, conceptually as a form of communication, has no uses. It's just that email as it stands today sorely needs refactoring to include everything from IMs (multimedia based or just pure text) to long form correspondence with varying degrees of confidentiality (encryption) and possibly including some form of file-sharing linking (not so much directly tied into the client/server protocols). I'm not sure the current email server and client protocols can handle such a refactoring without breaking backwards compatibility (therefore pissing off everyone who is still using email). Some email clients do some of the things I'm already suggesting. Some of them do it very well, but honestly am I off the mark on this line of thought? I have to think possibly I am, but I'll leave it to you all to consider.

</comment><comment><USER24> Can someone here comment on why creating & maintaining a solid local desktop email client is such a costly, moving target, such that it has required an organization the size of Mozilla to pursue it and get it "nearly" right? Is the bulk of Thunderbird maintenance work happening in its "HTML render/js component(s)"? If so, could we stabilize on a Thunderbird without HTML render/js support, providing it only as optional add-on?

<comment><USER10> Because email is a decades old technology that, thanks to its decentralization and such a wide spread usage, is impossible to improve.

I highly recommend this blog post about a guy trying to make its own email client and failing to do so miserably because emails are such a mess : [http://www.slate.com/articles/technology/technology/2015/02/...](http://www.slate.com/articles/technology/technology/2015/02/email_overload_building_my_own_email_app_to_reach_inbox_zero.html)

Edit: removed the Pocket redirect from the URL.

</comment></comment><comment><USER12> Related:

KDE is not the right place for Thunderbid

[http://www.elpauer.org/2016/04/kde-is-not-the-right-place- fo...](http://www.elpauer.org/2016/04/kde-is-not-the-right-place-for- thunderbid/)

<comment><USER25> Seems to have been deleted?

</comment></comment><comment><USER26> I guess the main question is who will actually own the Thunderbird license after the split? If the ownership of most of the code remains at Mozilla but Mozilla doesn't care, what would be the motivation of any funded entity to continue the development of Thunderbird?

<comment><USER27> to create a business model like airmail, but for the android counterpart. It would be cool!

</comment></comment><comment><USER28> thunderbird used to be a great mail client, but became extremely sluggish and I think it's also bloated. So I moved to claws-mail and mutt, they are lighting fast.</comment>


A good ppt on the media and it's tech phobias - <AUTHOR> http://www.mindhacks.com/blog/2009/07/fringe_benefits.html

<comment><USER2> I'm sorry to do this, but I can't resist.

Oh shit! Here comes an s!!!</comment>


PEP 3148 Accepted: futures – execute computations asynchronously - <AUTHOR> http://jessenoller.com/2010/07/11/pep-3148-accepted-futures-execute-computations-asynchronously/

<comment><USER2> Finally we get a parallel map!

<comment><USER3> You mean like: [http://docs.python.org/library/multiprocessing.html#multipro...](http://docs.python.org/library/multiprocessing.html#multiprocessing.pool.multiprocessing.Pool.map)</comment></comment>


Self-sufficiency is the road to poverty - <AUTHOR> http://www.johndcook.com/blog/2010/02/17/self-sufficiency-is-the-road-to-poverty/ <comment><USER2> <http://en.wikipedia.org/wiki/North_korea>

<http://en.wikipedia.org/wiki/Juche>

We can see this in action.</comment>