@waldechristian: "On Win32 Perl automatically adds the :crlf layer to a number of filehandles.
This causes a great many problems for a lot of Perl code, since constructs like $fh->binmode; and $s =~ s/\r//g; have to be added everywhere just in case that piece of perl would ever be executed on Windows. Usually this kind of thing only breaks tests, since they compare multiline text quite often; but every once in a while this breaks something important, like the CGI handler in Plack.
I've spent many a month now chasing these things and prodding CPAN authors into fixing them with great success, increasing the amount of CPAN distributions that install and work safely on Win32.
But i am dissatified and quite frankly sick of it. The reason is that as far as i can tell the perl modules are not wrong, not broken. Neither is Win32. In my perception i am chasing tails here to fix a bug in perl core by having everyone BUT perl core implement workarounds for the bug.
I have been working with Perl and Win32 for a long time. However i have never seen a single instance where this :crlf auto-adding was useful. The only and single binary code i know about that actually cares about this thing is notepad.exe.
I am entirely confident that if i were to smoke the entirety of CPAN with the latest perl; and then resmoke it with a fork of that perl, with :crlf auto-adding removed, the overall PASS count would increase a lot and many dists would automagically start testing/working right. Only a very miniscule amount of dists would fail, specifically those that are win32-only and do multiline text comparisons with win32-generated data, which is easily fixed and much smaller in scope than the current situation.
However i might be overlooking something.
As such i have to ask: Are there even any reasons in this day and age to retain this malicious functionality?
With regards,
Christian Walde"
    @waldechristian: "<quote>
The point was that in order to work right on windows you HAVE to add it it in many situations where it would otherwise be completely unnecessary. Yes, it has other uses and i'm not saying that binmode is bad. The cargo-culting of binmode that :crlf-autoadding FORCES is bad.
Sometimes you have access to the file handle, sometimes you're dealing with another module written by someone else that doesn't have binmode in there. In that case you need to strip them manually while you wait (days, weeks, months) for the CPAN author to accept your patch and upload a new release.
With regards,
Christian Walde"
    @craigaberry: "On Tue, Jun 28, 2011 at 1:26 PM, Christian Walde
<quote>
How about a test somewhere in between doing nothing and creating world
peace?  I don't fully understand the problem you're trying to solve,
but why not actually try a build of Perl and a few modules after
adding Win32 to the list of platforms that should not be considered
DOSish and thus not get the crlf layer pushed by default?  5 minutes
with the code suggests something like the following might be involved,
though it wouldn't surprise me if there were more to it than that.  In
any case, someone might learn something, and it might even be me
(though I'm happy to remain ignorant on this particular topic).
- perl.h;-0   2011-06-16 19:06:09 -0500
+++ perl.h      2011-06-28 14:09:12 -0500
@@ -5963,17 +5963,17 @@ int flock(int fd, int op);
 #ifndef O_TEXT
 #  define O_TEXT 0
 #endif
 #if O_TEXT != O_BINARY
     /* If you have different O_TEXT and O_BINARY and you are a CLRF shop,
      * that is, you are somehow DOSish. */
 #   if defined(__BEOS__) || defined(__HAIKU__) || defined(__VOS__) || \
-       defined(__CYGWIN__)
+       defined(__CYGWIN__) || defined(WIN32)
     /* BeOS/Haiku has O_TEXT != O_BINARY but O_TEXT and O_BINARY have
no effect;
      * BeOS/Haiku is always UNIXoid (LF), not DOSish (CRLF). */
     /* VOS has O_TEXT != O_BINARY, and they have effect,
      * but VOS always uses LF, never CRLF. */
     /* If you have O_TEXT different from your O_BINARY but you still are
      * not a CRLF shop. */
 #       undef PERLIO_USING_CRLF
 #   else"
    @waldechristian: "<quote>
There are endless permutations of this kind of thing, combining reading, writing, pre-testing, during testing, without binmode on various operations or with it on various operations. But I'll just show you one very simple and often seen example of this kind of behavior:
<url>
On Linux all tests pass because it uses :raw by default. On Windows however only the binmode loader works correctly.
So, yes, it does only need to be added for windows' benefit in order to fix misbehavior of perl core, with no appreciable newline effect on *nix.
Honestly, i wasn't aware of them and looked them up:
-C seems to have nothing at all to do with this, as it deals with unicode. Please correct me if i overlooked anything. If i am right: This newline issue has nothing to do with unicode and unicode has no business being in this amber63@example.net.
open.pm and PERLIO look neat. However, at the end of the day they are bandaids that are just slightly more convenient to use than binmode() and i'd still have to pester CPAN authors to cargocult them into their code to fix perl modules globally.
I missed this earlier and i would just like to mention that this is not completely true, since STDOUT/ERR/IN are also affected by this mangling.
I disagree, but quite honestly, i don't have the patience to dig up or make examples of situations where this breaks. More importantly though, it distracts from the actual point of the discussion:
Are there any reasons why :crlf auto-adding is still necessary nowadays?
With regards,
Christian Walde"
    @waldechristian: "<quote>
I'd be happy to and have done massive regression testing of literally all of CPAN before, when i was adding new things to EUMM, but i have to admit my ignorance in C: I have no idea how to compile perl for Windows on my own.
Sorry. :/
With regards,
Christian Walde"
    @fawaka: "On Tue, Jun 28, 2011 at 10:13 PM, Christian Walde
<quote>
er
No, Perl on Unix does not use =C2=AB:raw=C2=BB by default, it does
:=C2=ABunix:perlio=C2=BB. The fact that :raw is an identity operator op top=
 of
that stack doesn't mean :raw is involved in any way.
.
It's all about PerlIO stacks. -C can add :utf8 to the stack, and that
needs to be popped off too when opening binary files. :crlf isn't the
only non-binary layer ;-).
ke
The issue may be bigger on Windows than on Unix, but that doesn't mean
it isn't present on Unix.
Because it's the correct thing to do on DOSish platform?
Leon"
    @xdaveg: "On Tue, Jun 28, 2011 at 4:15 PM, Christian Walde
<quote>
See all of CSJEWELL's Perl-Dist-* distributions on CPAN.
FWIW, I'm with Christian on the broader point.  We shouldn't have
OS-specific filters.  It assumes something about how end-users want
data handled that may or may not be true.  The default should be raw
 and then users should be explicit about what encoding/filters they
want.
I'd like to see :crlf on Win32 deprecated in 5.16.
 David"
    @fawaka: "<quote>
Actually, :crlf is way more gross than that. It's behavior is very
much platform dependent. On Unix, pushing a :raw on top of a :crlf
pops it off the stack. On Windows on the other hand it leaves in on
the stack but disables translation by unsetting a flag. This is
because :crlf double-serves as a buffering layer.
Even more surprisingly, on windows :unix is always used as the
base-layer because :win32 is an unfinished mess.
I'd like to see :crlf become a pure translational layer that doesn't
inherit from :perlio. To get the current behavior, you'd need a
=C2=AB:unix:perlio:crlf=C2=BB stack, instead of the current =C2=AB:unix:crl=
f=C2=BB stack.
And if someone with actual knowledge of the Win32 API (that is
certainly not me) steps up that could even be =C2=AB:win32:crlf=C2=BB.
Leon"
    @waldechristian: "<quote>
Ok, i wasn't aware of what they do. Thanks for the correction.
I'd like to still emphasize though that i meant: Neither on Linux nor on Unix does Perl do any newline mangling. Only on Windows. And i struggle to find any reason for why it still needs to be there.
Yes, i understand that. But i am talking only and exclusively about newline transformation, since :utf8 isn't forced on anyone just because they have a specific os.
*nix has automatic newline transformation?
As we discussed on IRC, this means: Because it's tradition. And you agreed there, tradition is not a valid reason to keep bad behavior. :)
I have to say though: If anyone can explain why it became a tradition, that would be useful information in this discussion.
With regards,
Christian Walde"
    @fawaka: "On Tue, Jun 28, 2011 at 11:08 PM, Christian Walde
<quote>
Terminals, the teleprinter kind. Carriage return would reset the head
of the printer to the first column, line feed would scroll it down one
notch. Together they had the newline effect. CP/M was apparently
designed for such devices. DOS took over this convention from CP/M for
reasons of backwards compatibility. Windows did the same with DOS.
OK, I think I'm convinced in your favor now ;-).
Leon"
    @waldechristian: "<quote>
Haha, thanks a lot. I think that actually makes a very strong point towards how obsolete this behavior is. :)
With regards,
Christian Walde"
    @craigaberry: "On Tue, Jun 28, 2011 at 4:08 PM, Christian Walde
<quote>
The beginnings of the code is here:
<<url>
Having a crlf layer was part of the original design of PerlIO's "line
disciplines" as they were called then, discussed at some length here:
<<url>
There was some question then of whether the crlf layer should be
default for Windows.  I don't know exactly when or where the decision
to do that was made, but I suspect it was solving some actual problem."
    @waldechristian: "<quote>
Thanks for digging!
Looking at the mails it was meant to be a standin for CRLF/LF auto-detection, which would be meant to prevent getting mixed newlines into files. I'd argue that that is not a valid reason to keep it around, since it's terrible at that due to the programmer having to detect or decide it manually anyhow AND the distinction being entirely useless due to only notepad caring about CR. :)
With regards,
Christian Walde"
    @jand: "<quote>
It is not just "a tradition", "\x0D\x0A" is the official line
termination sequence for text files on Windows. Supporting both text and
binary data streams is actually mandated by the C standard; it is just a
property of POSIX systems that they behave identically. I can't find a
free spec on the net right now, but you can read it paraphrased here:
<url>
Just wishing that every computer is a VAX, or every OS is a version of
Linux, or at least somewhat based on POSIX doesn't make it true.
Maybe switching all file handles to binary mode by default will make a
couple more CPAN modules pass their test suites on Windows (by hiding
that they don't implement proper text mode semantics). But it really
would be somewhat similar to enforcing that Perl can only be compiled
when sizeof(int) == sizeof(char*) because we want to be able to store
pointers in int variables. All computers are PDP 11s after all, or
should at least look like one. :)
But how does all of this help the poor Windows user who will want to
use Perl to process, you know, regular text files on Windows that have
*not* be written on a POSIX system, or by Perl 5.16? Most text files on
Windows will continue to have CR+LF line endings. So if we read them in
binary mode, they will still match /\r\n^z/. And neither chop, nor
chomp, nor s/\n// will get rid of that pesky '\r'. How is this
intuitive for a text processing tool? And how many existing scripts is
that going to break?
So running automated CPAN testing is going to mislead you about the
impact this change is going to have.
As you can probably guess by now, I don't think pretending that text
mode is the same as binary mode is the right thing to do.
Cheers,
-Jan"
    @jand: "<quote>
I already gave the tl;dr explanation in my previous reply.  Minimal sample:
C:\>echo 123|perl -E "say <> =~ /^\d+$/ ? 'number' : 'not'"
number
C:\>echo 123|perl -E "binmode(STDIN); say <> =~ /^\d+$/ ? 'number' : 'not'"
not
Cheers,
-Jan"
    @xdaveg: "<quote>
sample:
ot'"
To me, that's an argument for STD* to have :crlf by default, since on
Windows the console uses CRLF, but not an argument for "open" to apply
:crlf by default to any arbitrary file.
I sort of like the idea of removing :crlf by default *if* we provide a
:text option that people can use to be explicit that they want a text
mode transformation.
E.g.
    use v5.16;
    open my $bin, "<", "binary.dat";  # :raw even on Win32
    open my $bin, "<:text", "textual.txt";  # :text
What that really implies is that under v5.16, Perl no longer assumes
that text files are the primary type of file being manipulated or that
text files for use on the local machine are the primary type of file
being manipulated.
I might even go so far as to wonder whether under v5.16, open should
warn unless a layer is specified (or the open pragma is in effect):
    open my $bin, "<:raw", "binary.dat";  # explicit
I could see people finding that annoying (oh, no! more characters to
type!) but it might be the way to encourage good practices about being
explicit about expected encodings on input/output handles (e.g. utf8).
 I don't know.  Maybe that's too nanny-ish for Perl.
 David"
    @tchrist: "[Brian, skip to the bottom for one of Perl's Unicode deficiencies.]
"Jan Dubois" <email_address> wrote
   on Tue, 28 Jun 2011 23:24:01 PDT: 
<quote>
(Not that Linux is different from Unix in any meaningful way here. Perhaps
 they meant MacOS 9?)
I actually consider that a violation of tr18's RL1.6 on linebreak handling:
    <url>
    RL1.6 Line Boundaries
	To meet this requirement, if an implementation provides for
	line-boundary testing, it shall recognize not only CRLF, LF, 
	CR, but also NEL (U+0085), PS (U+2029) and LS (U+2028).
    Formfeed (U+000C) also normally indicates an end-of-line. 
    For more information, see Chapter 3 of [Unicode].
    These characters should be uniformly handled in determining logical
    line numbers, start-of-line, end-of-line, and arbitrary-character
    implementations. Logical line number is useful for compiler error
    messages and the like. Regular expressions often allow for SOL and
    EOL patterns, which match certain boundaries. Often there is also a
    "non-line-separator" arbitrary character pattern that excludes line
    separator characters.
    The behavior of these characters may also differ depending on
    whether one is in a "multiline" mode or not. For more
    information, see Anchors and Other "Zero-Width Assertions" in
    Chapter 3 of [Friedl].
    A newline sequence is defined to be any of the following:
	\u000A | \u000B | \u000C | \u000D | \u0085 | \u2028 | \u2029 | \u000D\u000A
    [...]
    It is strongly recommended that there be a regular expression
    meta-character, such as "\R", for matching all line ending characters
    and sequences listed above (e.g. in #1). It would thus be shorthand
    for:
	( \u000D\u000A | [\u000A\u000B\u000C\u000D\u0085\u2028\u2029] )
And indeed, Perl's \R works that way  in a regex.  So you can do this
sort of thing: (all perl examples I give below are tested under v5.14)
    @lines = split /\R/,  $buffer;
    @paras = split /\R+/, $buffer;
And you *can* rewrite your
    say <> =~ /^\d+$/ 
to read:
    say <> =~ /^\d+\R?\z/ 
But the main point of RL1.6 is that you shouldn't have to do that.
So I think this is a bug that needs fixing:
    % perl -E 'say "\r\n" =~ /^$/ ? "Yes" : "No"'
    No
    % perl -E 'say "\r\n" =~ /^\R\z/ ? "Yes" : "No"'
    Yes
Also, and perhaps more germane to this thread, I don't know how to set $/
to anything that will let me get a \R-terminated line back from readline(),
nor which allows chomp() to then work appropriately on that return.  I 
believe neither is currently possible.  
How to fix that?  I/O layer?  Extension of :crlf?
RL1.6 also says:
    Note: For some implementations, there may be a performance impact in
          recognizing CRLF as a single entity, such as with an arbitrary
          pattern character ("."). To account for that, an implementation
          may also satisfy R1.6 if there is a mechanism available for
          converting the sequence CRLF to a single line boundary character
          before regex processing.
I honestly don't know whether :crlf's so-called newline-munging,
the topic of this thread, satisfies that condition.
tom
PS: CRLF is also good place to point out the difference between legacy
    and extended grapheme clusters, since only the extended version 
    counts that two-character sequence as a single grapheme cluster:
      Legacy Grapheme Cluster:
          % perl -E 'say "\r\n" =~ /^\p{Grapheme_Base}\p{Grapheme_Extend}*\z/ ? "Yes" : "No"'
          No
      Extended Grapheme Cluster:
          % perl -E 'say "\r\n" =~ /^\X\z/ ? "Yes" : "No"'                          
          Yes
    Which are both of them further distinguished from:
      Erroneous Grapheme Cluster:
          % perl -E 'say "\r\n" =~ /^\PM\pM*\z/ ? "Yes" : "No"'
          No
    But that's a someone longer story, which doesn't need recounting here."
    @jand: "<quote>
It has *nothing* to do with the console, but with the fact that *text* files
on Windows have CRLF line endings.  Download a text/plain file with your
browser, or redirect the output of most any program to a file, and
you end up with CRLF line endings.
Most programs work correctly with just LF terminated lines because that
is what you would end up after removing the CR anyways.  This wouldn't
work on Mac OS (which we don't support anymore), so it is an accidental
feature, but we could claim that it is an application of "be strict in
what you emit, and liberal in what you accept".
I would like to have a :text layer; I did need it at least once before,
but can't remember the details anymore. But:
Assuming that Perl is primarily used to manipulate *binary* files
seems just wrong to me.
I'm objecting to this proposal because it's intention is to *hide*
the portability problems of Perl scripts as long as they only operate
on POSIX style text files.  It does not encourage writing portable
scripts that work with *both* native and foreign line-endings.
If you want Cygwin functionality, you can find it over -> there.
The native port is supposed to work as best as possible with the
native OS. 
I think this is better done by Perl888a:dbb7:2d49:b3f6:8605:bf4e:caed:e5a4ritic.
As for encodings, they are completely orthogonal to the question of
line endings, and you could even argue that any character encoding
should also imply :text.  It would certainly help with this kind of
abomination, which is currently necessary to open a platform-native
text file in Unicode encoding on Windows:
  open(my $fh, ">:raw:encoding(UTF-16LE):crlf", $filename) or die;
Cheers,
-Jan"
    @tchrist: "<quote>
Would you believe that some programs' "Save as text" under Darwin still
produce "\r" for line separators?  It's true.  So you can get a 
webpage that you Save As text and it has no newlines in it at all!!
Sometimes it's even in MacRoman.  Yes, this is under Darwin.
-tom"
    @xdaveg: "<quote>
I think the original sin was assuming that text mode should be the
default.  Assuming binary is just as arbitrary, but it's arbitrary in
a way that does not change what is read or written, which I think is a
less intrusive assumption.  Obviously, I would only want that change
under
a "use v5.X" semantic rule.
I don't understand your argument.  I think people are making
assumptions either way.  My thought is that if code is protected by a
lexical version declaration, then we can expect that (with some
learning curve) they understand what assumptions are in force.
Very possibly.  That's another way to address the issue.
Ick.
My broader point was that now that (Unicode) encoded files are
increasingly common, it's not unreasonable to start nudging
programmers to be explicit about what transformations they want when
reading from a filehandle.  In that sense, the encoding isn't
orthogonal  both result in transformations of the octet-stream being
read/written.
 David"
    @jand: "<quote>
Assuming text mode is not arbitrary; it is optimizing for the
common use case.  Perl does a lot of that. :)
With text mode being the default, most code dealing with text is
cross-platform by default too.  With binary mode the default, programs
dealing with text files are platform-dependent unless you make an
extra effort to make them portable.
I'm just arguing that most *users* of Perl are processing text files.
The number of people writing network protocol layer code is comparatively
small, so let's have defaults that work for the majority and not just
for "us".
Indeed.  That is still no reason to break default text mode handling
on Windows when no layers are specified at all though. Well, IMO of
course.
Cheers,
-Jan"
    @tchrist: "[Brian: here's another Perl Unicode glitch or two for your bag.  Whether
        it is a bug I don't know, but RL1.2a does disagree with our \s.]
Ed Avis <email_address> wrote on Wed, 29 Jun 2011 13:46:57 -0000:
<quote>
This is the RL1.6 bug.  We have to fix it, somehow, although I don't
know how best to do that given all this newline-mangling discussion.
Different approaches are readily conceivable here, plus the spec is
somewhat fuzzy.
But that's not the same.  At all.  This is what \R was *made* for:
just use /\R?\z/ for /$/ and you will be fine (excepting the 5-for-1
Huffman failure, I suppose). 
Also, I'm a bit leery of \s these days.  Remember that \s is not the same
as [\h\v] — which if you didn't know about already, you almost certainly
don't want to. ;-{
tom"
    @tchrist: "Ed Avis <email_address> wrote on Wed, 29 Jun 2011 13:58:30 -0000: 
<quote>
And on Darwin, it gets CR alone, which is incredibly annoying.
tom"
    @ambrus: "On Tue, Jun 28, 2011 at 11:08 PM, Christian Walde
<quote>
Not with amber63@example.net.  Perl on Linux still changes crlf to lf in
perl source code it reads, such as in quoted strings and here-docs.
Whether this is good or bad is unclear, but it's probably related to
how many web servers when serving text files (such as perl source
code) transform lf to crlf, but most web browsers on Linux don't
translate crlf to lf when saving a downloaded text file.  (Somehow
some people named crlf the "network line separator" and big-endian the
"network byte order", and now people believe that mandates that they
must use those whenever they send anything over a network?  Okay,
sorry for the rant.)
Ambrus"
    @fawaka: "<quote>
g:
 provides for
only CRLF, LF,
(U+2028).
logical
acter
error
OL and
 also a
es line
n
n
u2028 | \u2029 | \u000D\u000A
n
haracters
orthand
85\u2028\u2029] )
his
),
=A0I
A PerlIO layer doing normalization would be the most obvious approach,
though it's not entirely clear how such a layer should behave on
output.
Leon"
    @tchrist: "Leon Timmermans <email_address> wrote on Thu, 30 Jun 2011 15:16:39 +0200: 
<quote>
One of the purported blessings (I find it a curse) of the
standard Java runtime library is that you when you do what is
essentially
    while read a line
	print a line
You *do* get a line back no matter what sort of line-ending it had.
The gotcha is that that you cannot really distinguish what you
started with.  It autochomps but won't tell you what it did.  So
you can't tell whether you chomped 0, 1, or 2 characters, nor
what if any those characters really were.
    <url>
That is, their readLine() method accepts any valid line-ending,
but its corresponding println() method only emits lines with 
the "native" line terminator added to them.   That must have
sounded good to somebody once upon a time (and can even be argued
to be following Postel's Law), but I find it a royal pain.
It's for line-terminator normalalization, not for clean copying.
May we please not go there? :(
I still want us to work with \R though.  I'm sorry I expressed it has a
pattern, because that has led to what sounds like a need for support of
arbitrary regex terminator.  I certainly didn't mean that.  You can
certainly implement the *equivalent* to a \R terminator in a non-regex way
without losing a great deal of efficiency.
tom"
    @rurban: "2011/6/28 Christian Walde <email_address>:
<quote>
Hello Christian,
Can you show us those fixes you needed in some modules please?
perl usually does the right thing, but in your case in certain cases
obviously not.
I've read the whole thread, and I don't believe we should generally
strip :crlf on Win32.
Testfiles and the console are :crlf on Windows generally, even if some
windows editors
nowadays can also read \n linefeeds alone. http headers are also \r\n,
but bodies generally not, text/plain on windows browsers maybe.
I assume http or net streams on windows should not to :crlf
translation, only file and console streams, but it might be more
complicated.
Reini Urban"
    @pagaltzis: "* Konovalov, Vadim (Vadim)** CTR ** <email_address> [2011-06-30 15:55]:
<quote>
At $work we have had huge headaches trying to keep CRLFs out of
our Git repositories. Sources that come out of Git use LF and the
editors used by Windows-based developers DTRT with those, but
files newly created by these developers often get checked in with
CRLF line endings.
To me that means most Windows software must still be presuming
CRLF even if it can handle LF-only properly.
Regards,
Aristotle Pagaltzis // <<url>"
    @ilmari: "Aristotle Pagaltzis <email_address> writes:
<quote>
Have a look at the core.autocrlf config setting (I think the value you
want is "input").
ilmari
"A disappointingly low fraction of the human race is,
 at any given time, on fire." - Stig Sandbeck Mathisen"
{'source': 'USENET', 'other_information': '{"reply_count": 28, "forum_name": "perl.perl5.porters.20150204.mbox"}', 'file': 'data/output/usenet-perl.perl5.parquet'}