Why do I have to type ctrl-d twice?
Question: For my own amusement, I've cooked up a python script that allows me to use
python for bash one-liners; Supply a python generator expression; and the
script iterates over it. Here's the script:

    
    
    DEFAULT_MODULES = ['os', 're', 'sys']
    
    _g = {}
    for m in DEFAULT_MODULES:
        _g[m] = __import__(m)
    
    import sys
    sys.stdout.writelines(eval(sys.argv[1], _g))
    

And here's how you might use it.

    
    
    $ groups | python pype.py '(l.upper() for l in sys.stdin)'
    DBORNSIDE
    $ 
    

For the intended use, it works perfectly!

But when I don't feed it with pipe and just invoke it directly, for instance:
[emphasis added to show what I type]

    
    
    $ python pype.py '("%r\n" % (l,) for l in sys.stdin)'
    **foo Enter
    barEnter
    bazEnter
    Ctrl DCtrl D**'foo\n'
    'bar\n'
    'baz\n'
    $ 
    

In order to stop accepting input and produce any output, I have to type either
`Enter` \- `Ctrl D` \- `Ctrl D` or `Ctrl D` \- `Ctrl D` \- `Ctrl D`. This
violates my expectations, that each line should be processed as entered, and
that typing `Ctrl D` at any time will end the script. Where is the gap in my
understanding?

EDIT: I've updated the interactive example to show that I'm not seeing the
quoting wim describes in his answer, and some more examples too.

    
    
    $ python pype.py '("%r\n" % (l,) for l in sys.stdin)'
    **foo Ctrl DCtrl DbarEnter
    Ctrl DCtrl D**'foobar\n'
    $ python pype.py '("%r\n" % (l,) for l in sys.stdin)'
    **foo Ctrl VCtrl D**^D**bar Enter
    Ctrl DCtrl D**'foo\x04bar\n'
    $ 
    


Answer: `Ctrl-D` is recognized not necessarily as EOF, but as "terminate current
`read()` call".

If you have an empty line (or just pressed `Ctrl-D`) and press `Ctrl-D`, your
`read()` terminates immediately and returns 0 read bytes. And this is a sign
for EOF.

If you have data in a line and press `Ctrl-D`, your `read()` terminates with
whatever there has been typed, of course without a terminating newline
(`'\n'`).

So if you have input data, you press `Ctrl-D` twice of a non-empty line or
once on a empty one, i.e. with `Enter` before.

This all holds for the normal OS interface, accessible from Python via
`os.read()`.

Python file objects, and also file iterators, treat the first EOF recognized
as termination for the current `read()` call, as they suppose there is nothing
any longer. A next `read()` call tries again and needs another `Ctrl-D` in
order to really return 0 bytes. The reason is that a file object `read()`
always tries to return as many bytes as requested and tries to fill up if a OS
`read()` returns less than requested.

As opposite to `file.readline()`, `iter(file)` uses the internal `read()`
functions to read and thus always has this special requirement of the extra
`Ctrl-D`.

I always use `iter(file.readline, '')` to read line-wise from a file.

