[Tutor] Yielding from a with block

[Oscar Benjamin]

import csv

I'm just wondering what other people think about this. Should code
like make_lines below be discouraged?

yield htbegin
yield from fprint(linelist)
yield htend
yield ("<td class=l>{}</td><td>{}</td><td>{}</td>"
"<td>{}</td><td>{}</td><td>{}</td>\n").format(*linelist)
yield '</tr>\n'

There's a fundamental contradiction between the with and yield
statements. With means "don't exit this block without running my
finalisation routine". Yield means "immediately exit this block
without doing anything (including any finalisation routines) and then
allow anything else to occur".

Using yield inside a with statement like this renders the with
statement pointless: either way we're really depending on __del__ to
execute the finalisation code. So it with or without the with
statement it will work fine on CPython. On other implementations it
may or may not close the file with or without the with statement. IOW
the with statement is not able to provide the guarantees normally
expected of it when used this way.

It's usually fairly trivial to rearrange things so that this doesn't happen:

def wrap_header_footer(fin):
yield htbegin
for linelist in csv.reader(fin):
yield from fprint(linelist)
yield htend


--
Oscar
_______________________________________________
Tutor maillist - ***@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Cameron Simpson]

Post by Oscar Benjamin

import csv

I'm just wondering what other people think about this. Should code
like make_lines below be discouraged?

In my opinion, no.

Post by Oscar Benjamin

yield htbegin
yield from fprint(linelist)
yield htend
yield ("<td class=l>{}</td><td>{}</td><td>{}</td>"
"<td>{}</td><td>{}</td><td>{}</td>\n").format(*linelist)
yield '</tr>\n'

There's a fundamental contradiction between the with and yield
statements. With means "don't exit this block without running my
finalisation routine". Yield means "immediately exit this block
without doing anything (including any finalisation routines) and then
allow anything else to occur".

Not really. Yield passes a return value back to the iterator user and _pauses_
execution of the generator. The generator's execution context is _still_ inside
the "with".

Certainly if the user/consumer of the generator does not consume all the way to
the end then the "with" exit code won't fire. At least in a timely fashion;
might it fire during the cleanup phase? I'd think so.

However, the common circumstance is that the consumer does run to the end of
the iterator, and therefore the "with" exit process _will_ run when because
that is a standard part of the generator's execution.

BTW, I think including the fprint() function here is a furphy; it doesn't
affect the issue discussed.

Post by Oscar Benjamin
Using yield inside a with statement like this renders the with

This is not true.

Post by Oscar Benjamin
either way we're really depending on __del__ to
execute the finalisation code. So it with or without the with
statement it will work fine on CPython. On other implementations it
may or may not close the file with or without the with statement. IOW
the with statement is not able to provide the guarantees normally
expected of it when used this way.

If make_lines() above runs to completion then the "with" exit process will also
run and everything will be fine.

It is only when make_lines() is not completely consumed that the file may not
be closed in a timely fashion.

Why? Because the consumer does not get to see end-of-iterator until
make_lines() actually returns, and that directly implies completion of the
"with" exit phase.

Post by Oscar Benjamin
yield htbegin
yield from fprint(linelist)
yield htend

I would not encourage this kind of thing except in quite special circumstance.
For your example, failure to completely consume make_ines() will result in
invalid HTML output (missing closing "htend", for example). So the result of
the entire process will be invalid anyway, with the leaked open file just
another side effect.

Normally (ignoring unbounded iterators), the same scenario will apply: you
always iterate to the end of somehting like your example because to do
otherwise will usually result in an invalid/incorrect final result. So while
technically the risk you describe exists, in practice it will almost never
occur unless the larger program outcome also results in failure of some kind.

Cheers,
Cameron Simpson <***@zip.com.au>

I object to doing things that computers can do. - Olin Shivers
_______________________________________________
Tutor maillist - ***@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Steven D'Aprano]

Post by Oscar Benjamin
I'm just wondering what other people think about this. Should code
like make_lines below be discouraged?

yield htbegin
yield from fprint(linelist)
yield htend

[...]

Post by Oscar Benjamin
There's a fundamental contradiction between the with and yield
statements. With means "don't exit this block without running my
finalisation routine". Yield means "immediately exit this block
without doing anything (including any finalisation routines) and then
allow anything else to occur".

No, I think you have misunderstood the situation, and perhaps been
fooled by two different usages of the English word "exit".


Let me put it this way:

with open(somefile) as f:
text = f.write() # here
...

In the line marked "here", execution exits the current routine and
passes to the write method. Or perhaps it is better to say that
execution of the current routine pauses, because once the write method
has finished, execution will return to the current routine, or continue,
however you want to put it. Either way, there is a sense in which
execution has left the with-block and is running elsewhere.

I trust that you wouldn't argue that one should avoid calling functions
or methods inside a with-block?

In this case, there is another sense in which we have not left the
with-block, just paused it, and in a sense the call to write() occurs
"inside" the current routine. Nevertheless, at the assembly language
level, the interpreter has to remember where it is, and jump to the
write() routine, which is outside of the current routine.

Now let us continue the block:

with open(somefile) as f:
text = f.write()
yield text # here


Just like the case of transferring execution to the write() method, the
yield pauses the currently executing code (a coroutine), and transfers
execution to something else. That something else is the caller. So in
once sense, we have exited the current block, but in another sense we've
just paused it, waiting to resume, no different from the case of
transferring execution to the write() method.

In this case, the with block is not deemed to have been exited until
execution *within the coroutine* leaves the with-block. Temporarily
pausing it by yielding leaves the coroutine alive (although in a paused
state), so we haven't really exited the with-block.

Post by Oscar Benjamin
Using yield inside a with statement like this renders the with
statement pointless: either way we're really depending on __del__ to
execute the finalisation code.

No, that's not correct. __del__ may not enter into it. When you run the
generator to completion, or possibly even earlier, the with-statement
exits and the file is closed before __del__ gets a chance to run.
__del__ may not run until Python exits, but the file will be closed the
moment execution inside the generator leaves the with-statement.
Consider:

def gen():
with open(somefile) as f:
yield "Ready..."
yield "Set..."
yield "Go!"

it = gen()
next(it)

At this point, the generator is paused at the first yield statement, and
the file is held open, possibly indefinitely. If I never return control
to the generator, eventually the interpreter will shut down and call
__del__. But if I continue:

next(it) # file closed here

the with-block exits, the file is closed, and execution halts after the
next yield. At this point, the generator is paused, but the file object
is closed. The garbage collector cannot clean up the file (not that it
needs to!) because f is still alive. If I never return control to the
generator, then the garbage collector will never clean up the reference
to f, __del__ will never run, but it doesn't matter because the file is
already closed. But if I continue:

next(it)
next(it)

we get one more value from the generator, and on the final call to
next() we exit the generator and f goes out of scope and __del__ may be
called.

[Oscar Benjamin]

Post by Steven D'Aprano

Post by Oscar Benjamin
I'm just wondering what other people think about this. Should code
like make_lines below be discouraged?

yield htbegin
yield from fprint(linelist)
yield htend

[...]

Post by Oscar Benjamin
There's a fundamental contradiction between the with and yield
statements. With means "don't exit this block without running my
finalisation routine". Yield means "immediately exit this block
without doing anything (including any finalisation routines) and then
allow anything else to occur".

No, I think you have misunderstood the situation, and perhaps been
fooled by two different usages of the English word "exit".
text = f.write() # here
...
In the line marked "here", execution exits the current routine and
passes to the write method. Or perhaps it is better to say that
execution of the current routine pauses, because once the write method
has finished, execution will return to the current routine, or continue,
however you want to put it. Either way, there is a sense in which
execution has left the with-block and is running elsewhere.
I trust that you wouldn't argue that one should avoid calling functions
or methods inside a with-block?
In this case, there is another sense in which we have not left the
with-block, just paused it, and in a sense the call to write() occurs
"inside" the current routine. Nevertheless, at the assembly language
level, the interpreter has to remember where it is, and jump to the
write() routine, which is outside of the current routine.
text = f.write()
yield text # here
Just like the case of transferring execution to the write() method, the
yield pauses the currently executing code (a coroutine), and transfers
execution to something else. That something else is the caller. So in
once sense, we have exited the current block, but in another sense we've
just paused it, waiting to resume, no different from the case of
transferring execution to the write() method.
In this case, the with block is not deemed to have been exited until
execution *within the coroutine* leaves the with-block. Temporarily
pausing it by yielding leaves the coroutine alive (although in a paused
state), so we haven't really exited the with-block.

I'm sure you understand the fundamental difference between calling a
function and yielding inside a with statement: when calling a function
the new frame is *appended* to the call stack keeping the current
frame and all of its exception traps and context managers in place
ready for unwinding. When yielding the current frame is *removed* from
the call stack (along with its exception traps and context managers).

When a with block is used without a yield it is not possible (barring
unrecoverable failure of the runtime itself) to leave the block
without calling its finaliser. It doesn't matter if we call a function
that in turn calls a generator. As long as there is no yield inside
the with block in the current frame then the finalisation guarantee is
maintained. When using yield we don't have this guarantee and in fact
there are common non-exceptional cases (break/return) where the
undesirable occurs.

Post by Steven D'Aprano

Post by Oscar Benjamin
Using yield inside a with statement like this renders the with
statement pointless: either way we're really depending on __del__ to
execute the finalisation code.

No, that's not correct. __del__ may not enter into it. When you run the
generator to completion, or possibly even earlier, the with-statement
exits and the file is closed before __del__ gets a chance to run.

[snip]

Obviously there is this case in which the with achieves its purpose
but I mean this in a more general context than the specific example. I
consider that it is generally best to avoid doing this. In the same
way that I would advise someone to always use with when working with
files (even though it often doesn't matter) I would advise someone
generally not to yield from a with statement.


--
Oscar
_______________________________________________
Tutor maillist - ***@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Steven D'Aprano]

Post by Oscar Benjamin
I'm sure you understand the fundamental difference between calling a
function and yielding inside a with statement: when calling a function
the new frame is *appended* to the call stack keeping the current
frame and all of its exception traps and context managers in place
ready for unwinding. When yielding the current frame is *removed* from
the call stack (along with its exception traps and context managers).

Implementation details.

Post by Oscar Benjamin
When a with block is used without a yield it is not possible (barring
unrecoverable failure of the runtime itself) to leave the block
without calling its finaliser.

Yes.

Post by Oscar Benjamin
It doesn't matter if we call a function
that in turn calls a generator.

This sentence is ambiguous. Do you mean "calls a generator function", or
do you mean "call [next() on] a generator object"?

I don't suppose it really matters though.

Post by Oscar Benjamin
As long as there is no yield inside
the with block in the current frame then the finalisation guarantee is
maintained. When using yield we don't have this guarantee and in fact
there are common non-exceptional cases (break/return) where the
undesirable occurs.

Er, no. You still have the finalisation guarantee. You just have to
understand what the guarantee actually is. It does *not* refer to
pausing the generator to pass control back to the caller. That would
make yield inside a with block absolutely useless.

The guarantee is that when you *exit* (not pause) the with block, then
and only then will the context manager's __exit__ method run. That's no
different from the non-generator use of a CM.

If you think that guarantee is not made, you have to either demonstrate
a counter-example where exiting the with block fails to run the __exit__
method, or point us to official documentation supporting your position.

Otherwise I stand by my earlier position that you are misinterpreting
what it means to exit a with block. Pausing it to yield is not an exit.

I did an experiment, where I tried to break the finalisation
guarantee using break, return and raise:

class CM:
def __enter__(self):
return self
def __exit__(self, *args):
print("exiting")

def test(n):
for i in range(1):
with CM():
if n == "break": break
if n == "return": return
if n == "raise": raise RuntimeError
yield 1



Falling out the bottom of the generator finalises correctly. So do
break, return and raise.

it = test("")
x = next(it)
next(it, None) # prints "exiting"

it = test("break")
next(it, None) # prints "exiting"

it = test("return")
next(it, None) # prints "exiting"

it = test("raise")
try: next(it)
except: pass # prints "exiting"



Under what circumstances can execution leave the with block without the
finalisation method __exit__ running?

[Oscar Benjamin]

Post by Steven D'Aprano
Otherwise I stand by my earlier position that you are misinterpreting
what it means to exit a with block. Pausing it to yield is not an exit.
I did an experiment, where I tried to break the finalisation
return self
print("exiting")
if n == "break": break
if n == "return": return
if n == "raise": raise RuntimeError
yield 1
Falling out the bottom of the generator finalises correctly. So do
break, return and raise.
it = test("")
x = next(it)
next(it, None) # prints "exiting"
it = test("break")
next(it, None) # prints "exiting"
it = test("return")
next(it, None) # prints "exiting"
it = test("raise")
try: next(it)
except: pass # prints "exiting"
Under what circumstances can execution leave the with block without the
finalisation method __exit__ running?

The break/return should take place in the loop that controls the generator e.g.:

$ cat gencm.py

class CM:
def __enter__(self):
print("Entering")
return self
def __exit__(self, *args):
print("Exiting")

def generator():
with CM():
yield 1
yield 2
yield 3

g = generator()

def f():
for x in g:
break # Or return

f()

print("End of program")

$ python3 gencm.py
Entering
End of program
Exiting

The context manager was triggered by the end of the program. CPython
tries to call all the __del__ methods for all live objects at process
exit. Now run the same under pypy:

$ pypy --version
Python 2.7.2 (1.8+dfsg-2, Feb 19 2012, 19:18:08)
[PyPy 1.8.0 with GCC 4.6.2]

$ pypy gencm.py
Entering
End of program

The __exit__ method was not called at all under pypy. Even if I don't
keep a reference to g outside of f the __exit__ method is not called
under this version of pypy (I don't have another to test with).


--
Oscar
_______________________________________________
Tutor maillist - ***@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Steven D'Aprano]

Post by Oscar Benjamin
$ pypy --version
Python 2.7.2 (1.8+dfsg-2, Feb 19 2012, 19:18:08)
[PyPy 1.8.0 with GCC 4.6.2]
$ pypy gencm.py
Entering
End of program
The __exit__ method was not called at all under pypy. Even if I don't
keep a reference to g outside of f the __exit__ method is not called
under this version of pypy (I don't have another to test with).

I think that is a clear bug in pypy.

[Laura Creighton]

Post by Oscar Benjamin
Python 2.7.2 (1.8+dfsg-2, Feb 19 2012, 19:18:08)
[PyPy 1.8.0 with GCC 4.6.2]
$ pypy gencm.py
Entering
End of program
The __exit__ method was not called at all under pypy. Even if I don't
keep a reference to g outside of f the __exit__ method is not called
under this version of pypy (I don't have another to test with).
--
Oscar

I do. Same behaviour for PyPy 2.4 and PyPy 2.5.1 . The __exit __ method
isn't called.

Laura
_______________________________________________
Tutor maillist - ***@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Peter Otten]

Post by Oscar Benjamin
I'm just wondering what other people think about this. Should code
like make_lines below be discouraged?

yield htbegin
yield from fprint(linelist)
yield htend

In my opinion, yes.

As I've used the pattern since Python 2.5 when try...finally in generators
became legal I will need some time to unlearn.

Post by Oscar Benjamin
There's a fundamental contradiction between the with and yield
statements. With means "don't exit this block without running my
finalisation routine". Yield means "immediately exit this block
without doing anything (including any finalisation routines) and then
allow anything else to occur".
Using yield inside a with statement like this renders the with
statement pointless: either way we're really depending on __del__ to
execute the finalisation code. So it with or without the with
statement it will work fine on CPython. On other implementations it
may or may not close the file with or without the with statement. IOW
the with statement is not able to provide the guarantees normally
expected of it when used this way.

Even if you limit yourself to CPython there is another effect: the order of
execution may not meet one's expectations/requirements:

$ cat with_in_generator.py
import contextlib

@contextlib.contextmanager
def demo():
print("before")
try:
yield
finally:
print("after")

def gen(items="abc"):
with demo():
yield from items

if __name__ == "__main__":
g = gen()
for item in g:
print(item)
if item == "b":
break
print("bye")
$ python3 with_in_generator.py
before
a
b
bye
after
$

(in case you don't spot it: "after" should be printed before "bye")
To get a well-defined order of execution you can close the generator
explicitly

$ cat with_in_generator2.py
[...]
if __name__ == "__main__":
g = gen()
with contextlib.closing(g) as h:
for item in h:
print(item)
if item == "b":
break
print("bye")
$ python3 with_in_generator2.py
before
a
b
after
bye

...but the obvious route is of course

Post by Oscar Benjamin
yield htbegin
yield from fprint(linelist)
yield htend

which in this case also has the advantage of better separation of concerns
(I'd probably move the csv.reader() out of the generator, too).

PS: I'm still looking for a fairly elegant rewrite of the problematic

def lines(files):
for file in files:
with open(files) as f:
yield from f

(see Oscar's comment in
<https://mail.python.org/pipermail/tutor/2015-May/105448.html>)

_______________________________________________
Tutor maillist - ***@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Oscar Benjamin]

Post by Peter Otten
...but the obvious route is of course

yield htbegin
yield from fprint(linelist)
yield htend

which in this case also has the advantage of better separation of concerns
(I'd probably move the csv.reader() out of the generator, too).

Agreed. In practise you would at least want to be able to pass the
filename in and then it's almost always better to be able to pass in a
file object.

Post by Peter Otten
PS: I'm still looking for a fairly elegant rewrite of the problematic
yield from f

This was mentioned in the previous thread but a fileinput.input object
can be used as a contextmanager so:

import fileinput

filenames = 'f.py', 'wrap.py'

with fileinput.input(filenames) as joined:
for line in joined:
print(line, end='')

The behaviour of fileinput in the corner case of an empty file list
(reading from stdin) is annoying in this usage though.

You can also just wrap your lines generator in a context manager:

from contextlib import contextmanager

@contextmanager
def catfiles(filenames):
def lines():
for filename in filenames:
with open(filename) as fin:
yield from fin
gen = lines()
try:
yield gen
finally:
gen.close()

filenames = 'f.py', 'wrap.py'

with catfiles(filenames) as joined:
for line in joined:
print(line.upper(), end='')


--
Oscar
_______________________________________________
Tutor maillist - ***@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Steven D'Aprano]

Post by Peter Otten
Even if you limit yourself to CPython there is another effect: the order of

[snip example]

That's an interesting example, and I can't tell if that's a
problem with your (and my) expectations, or a bug in the context
manager implementation.

PEP 343 clearly warns that the finally clause may not run immediately:

Note that we're not guaranteeing that the finally-clause is
executed immediately after the generator object becomes unused

https://www.python.org/dev/peps/pep-0343/

but the documentation for the with-statement suggests strongly that the
__exit__ method will run immediately after the block is exited, before
any additional code (including before any exception is raised. E.g.:

5. The suite is executed.

6. The context manager’s __exit__() method is invoked. If an
exception caused the suite to be exited, its type, value, and
traceback are passed as arguments to __exit__(). Otherwise,
three None arguments are supplied.

https://docs.python.org/2/reference/compound_stmts.html#the-with-statement

Post by Peter Otten
PS: I'm still looking for a fairly elegant rewrite of the problematic
yield from f
(see Oscar's comment in
<https://mail.python.org/pipermail/tutor/2015-May/105448.html>)

Oscar's comment includes a syntax error, which makes it hard to run his
code:

print("__exit__ called")__del__.

I can't even begin to guess what that is supposed to be, and reading the
next few messages in the thread doesn't enlighten.

[Oscar Benjamin]

Post by Steven D'Aprano

Post by Peter Otten
Even if you limit yourself to CPython there is another effect: the order of

[snip example]
That's an interesting example, and I can't tell if that's a
problem with your (and my) expectations, or a bug in the context
manager implementation.

I think that this behaviour is consistent with the generally
unspecified language-level behaviour of deallocation and __del__.
Consider the language specification for this version of Python (and
note the last sentence in particular):

"""
object.__del__(self)

Called when the instance is about to be destroyed. This is also called
a destructor. If a base class has a __del__() method, the derived
class’s __del__() method, if any, must explicitly call it to ensure
proper deletion of the base class part of the instance. Note that it
is possible (though not recommended!) for the __del__() method to
postpone destruction of the instance by creating a new reference to
it. It may then be called at a later time when this new reference is
deleted. It is not guaranteed that __del__() methods are called for
objects that still exist when the interpreter exits.
"""
https://docs.python.org/2/reference/datamodel.html#object.__del__

Post by Steven D'Aprano
Note that we're not guaranteeing that the finally-clause is
executed immediately after the generator object becomes unused
https://www.python.org/dev/peps/pep-0343/

The rest of that sentence says "..., even though this is how it will
work in CPython." To me that clearly and deliberately permits other
implementations to behave differently.

Post by Steven D'Aprano
but the documentation for the with-statement suggests strongly that the
__exit__ method will run immediately after the block is exited, before
5. The suite is executed.
6. The context manager’s __exit__() method is invoked. If an
exception caused the suite to be exited, its type, value, and
traceback are passed as arguments to __exit__(). Otherwise,
three None arguments are supplied.
https://docs.python.org/2/reference/compound_stmts.html#the-with-statement

But the with statement is not "exited" as the suite is unfinished. The
generator has suspended itself and removed itself from the call-stack
so that it is no longer permitted to catch exceptions etc. Also note
that the removal from the call stack is not an implementation detail.
The interpreter can implement the call stack how it likes but it must
be semantically equivalent to a call stack with exception unwinding in
order to meet the general definition of the language.

Post by Steven D'Aprano

Post by Peter Otten
PS: I'm still looking for a fairly elegant rewrite of the problematic
yield from f
(see Oscar's comment in
<https://mail.python.org/pipermail/tutor/2015-May/105448.html>)

Oscar's comment includes a syntax error, which makes it hard to run his
print("__exit__ called")__del__.
I can't even begin to guess what that is supposed to be, and reading the
next few messages in the thread doesn't enlighten.

Some kind of editing error. It should be:

print("__exit__ called")


--
Oscar
_______________________________________________
Tutor maillist - ***@python.org
To unsubscribe or change subscription options:
https:

[Steven D'Aprano]

Post by Oscar Benjamin

Post by Steven D'Aprano

Post by Peter Otten
Even if you limit yourself to CPython there is another effect: the order of

[snip example]
That's an interesting example, and I can't tell if that's a
problem with your (and my) expectations, or a bug in the context
manager implementation.

I think that this behaviour is consistent with the generally
unspecified language-level behaviour of deallocation and __del__.

Yes it is. There's nothing unusual going on in this example. The
generator is still paused *inside the with block*, and it doesn't exit
the with block until the generator is closed on deallocation, which
might not be until interpreter exit.

[...]

Post by Oscar Benjamin
It is not guaranteed that __del__() methods are called for
objects that still exist when the interpreter exits.

Okay, well that gives pypy a "Get Out Of Jail Free" card for the
behaviour you demonstrated. I guess it's not necessarily a bug.


[...]

Post by Oscar Benjamin
But the with statement is not "exited" as the suite is unfinished.

Exactly! So Peter's expectation that it should be finalised is wrong. He
managed to convince me for a few moments that there was something
mysterious going on, but on further thought I realised that the observed
behaviour is correct and should be expected. Escaping from the "for x in
generator" loop early leaves the generator paused inside the with
block.

Post by Oscar Benjamin
The
generator has suspended itself and removed itself from the call-stack
so that it is no longer permitted to catch exceptions etc. Also note
that the removal from the call stack is not an implementation detail.
The interpreter can implement the call stack how it likes but it must
be semantically equivalent to a call stack with exception unwinding in
order to meet the general definition of the language.

The documention of with statements and context managers says nothing
about the call stack, and nor should it, since the call stack is part of
the implementation of the Python virtual machine, not part of the Python
runtime. What's important is not *how* a paused generator ceases to
catch exceptions etc. but the fact that it no longer does. That's what I
mean by calling it an implementation detail.

[Steven D'Aprano]

Post by Peter Otten
Even if you limit yourself to CPython there is another effect: the order of

No, wait, forget everything I said in my previous post. This is
*clearly* a case where our expectations were wrong, and the context
manager guarantee is working correctly. I was confused because I was
over-thinking it and seeing something unexpected when in fact it is
working exactly as promised.

Post by Peter Otten
$ cat with_in_generator.py
import contextlib
@contextlib.contextmanager
print("before")
yield
print("after")
yield from items
g = gen()
print(item)
break
print("bye")

Since you *break* from the for-loop, the generator g never runs to
completion. Since it is still paused *inside the with block*, naturally
the context manager __exit__ doesn't run. If it did run, THAT would be a
violation of the context manager guarantee!

Now that you have broken out of the for-loop, you still have a reference
to g, and are perfectly entitled to hang on to the reference for a
while, then iterate over it again, or directly call next(g). Until such
time as you do, or explicitly call g.close(), the context manager has to
stay open.
I

Until you do so (or call g.close() to explicitly end it), it is paused
inside the with block.

There's nothing to see here. The context manager is working correctly,
and if you expect it to __exit__ while still inside the with block, it
is your expectations that are wrong.

Post by Peter Otten
(in case you don't spot it: "after" should be printed before "bye")

That's mistaken. Just because you exit from the for-loop, doesn't mean
the generator is complete. Suppose you wrote a generator like this:

def gen():
yield 1
yield 2
yield 3
print("closing")

for i in gen():
if i == 2: break

print("bye")


Would you still expect it to print "closing" before "bye"?

[Mark Lawrence]

As interesting as this may be does it really belong on the tutor mailing
list?
--
My fellow Pythonistas, ask not what our language can do for you, ask
what you can do for our language.

Mark Lawrence

_______________________________________________
Tutor maillist - ***@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor