Python generators and yield // James Cooke // Brighton-based Python developer

It started with an interview

Last week in an interview for a Django developer job, I was asked:

thing = (x**2 for x in xrange(10))

What is the type of thing?

Although I was able to identify that the type is dependent on the () around the list-comprehension-like-construction, I didn’t know the exact type that thing would be.

The answer is a generator.

This post shows some of the functionalities of generators and how they can be used in Python control flow.

Generator expressions

Generators can be created with generator expressions. A generator expression is a bit like a list comprehension. List Comprehension uses square brackets []. In Python…

>>> [x**2 for x in range(10)]

[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

A generator expression is a shortcut that allows generators to be created easily with a similar syntax - this time it’s using parentheses ().

>>> (x**2 for x in range(10))

<generator object <genexpr> at 0x2fa5eb0>

Generators are iterators

Generators “provide a convenient way to implement the iterator protocol”.

In Python, an iterator provides two key functions, __iter__ and next, so a generator itself must provide these two functions:

>>> my_gen = (x**2 for x in range(10))
>>> my_gen.__iter__
<generator object <genexpr> at 0x293c3c0>

__iter__ is there and returns the generator, now for next…

>>> my_gen.next()
0
>>> my_gen.next()
1

Therefore next works. We can keep hitting until…

>>> my_gen.next()
81
>>> my_gen.next()
---------------------------------------------------------------------------
StopIteration                             Traceback (most recent call last)
<ipython-input-19-b28d59f370d8> in <module>()
----> 1 zzz.next()

StopIteration:

A StopIteration is raised - so the generator does everything we’d expect it to by the iterator protocol.

Building a generator with yield

Although it’s not clear from the example above, a generator is able to relinquish control and return a value - while saving its state. It then allows the control to pass back to the structure that called it, until it’s called again, picking up where it left off.

This allows for loops over sets of values to be programmed, without the full list of values being calculated first. A generator can be used so that next is called before each iteration required.

In this way, only the values required for each iteration need to be computed.

The yield keyword - simple example

Adding yield to a function allows for generators to be constructed ‘manually’.

At its very simplest, a function could be written just to generate a single value. However, to show that a generator can return to its previous state when called again, let’s make one with two values. For example…

def two_things():
    yield 1
    yield 'hi'

Now we can make an instance of the generator.

>>> my_things = two_things()
>>> my_things
<generator object two_things at 0x31d0960>

And we can ask for next value.

>>> my_things.next()
1

Now when we call next again, our generator continues from the state of the last yield.

>>> my_things.next()
'hi'

So you see how different values can be returned, one after the other.

And after that second thing, the generator now raises a StopIteration, since it has no further values to return.

Since a generator implements the iterator protocol, it can be used in a for statement and therefore in a list comprehension. This makes for a convenient way to check the values of a limited generator like this one.

>>> [x for x in two_things()]
[1, 'hi']

More complex example with yield

So let’s write Fibonacci as a generator. I’m going to start with doctests to create the definition of the function, then put the code at the end.

What I like about the doctests in this example is that in 3 fib is tested with next, but in 4 it’s tested using a list comprehension.

def fib(last):
    """

    1.  Creates a generator
    >>> type(fib(0))
    <type 'generator'>

    2.  fib(0) just generates 0th value (1)
    >>> zero_fib = fib(0)
    >>> zero_fib.next()
    1
    >>> zero_fib.next()
    Traceback (most recent call last):
    ...
    StopIteration

    3.  fib(1) creates a generator that creates 0th (1) and 1st (1) values of
        fib seq
    >>> one_fib = fib(1)
    >>> one_fib.next()
    1
    >>> one_fib.next()
    1
    >>> one_fib.next()
    Traceback (most recent call last):
    ...
    StopIteration

    4.  fib(10) generates the first 10 fibonacci numbers
    >>> [x for x in fib(10)]
    [1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89]

    """
    result = 1
    x = 0
    a = 1
    b = 0

    while x <= last:
        yield result

        result = a + b
        b = a
        a = result
        x += 1

That’s all - have fun with generators!