Python Basics: List Comprehensions, Dictionary Comprehensions and Generator Expressions

List comprehensions, dictionary comprehensions, and generator expressions are three powerful examples of such elegant expressions. In this post, we will take a look at for-loops, list comprehensions, dictionary comprehensions, and generator expressions to demonstrate how each of them can save you time and make Python development easier.

List and dictionary comprehensions in Python

To understand the basis of list and dictionary comprehensions, let’s first go over for-loops.

for-loops

In Python, a for-loop is perfect for handling repetitive programming tasks, as it can be used to iterate over a sequence, such as a list, dictionary, or string.

Let’s take a look at a simple example using a list:

words = ['cat', 'window', 'ball']

for x in words:

    print(x)

The result is each element printed one by one, in a separate line:

cat

window

ball

As you get to grips with more complex for-loops, and subsequently list comprehensions and dictionary comprehensions, it is useful to understand the logic behind them.

A for-loop works by taking the first element of the iterable (in the above case, a list), and checking whether it exists. If it does, the required action is performed (in the above case, print). The loop then starts again and looks for the next element. If that element exists the required action is performed again. This behaviour is repeated until no more elements are found, and the loop ends.

Using the range() function

The very useful range() function is an in-built Python function and is used almost exclusively with for-loops. Essentially, its purpose is to generate a sequence of numbers.

Let’s look at an example to see how it works:

for i in range(5):

       print(i)

The result is:

0

1

2

3

4

Be aware that the range() function starts from 0, so range(5) will return the numbers 0 to 4, rather than 1 to 5.

By default, the sequence will start from 0, increment in steps of 1, and end on a specified number. It is possible, however, to define the first element, the last element, and the step size as range(first, last, step_size).

List comprehensions

List comprehensions provide a more compact and elegant way to create lists than for-loops, and also allow you to create lists from existing lists.

List comprehensions are constructed from brackets containing an expression, which is followed by a for clause, that is [item-expression for item in iterator] or [x for x in iterator], and can then be followed by further for or if clauses: [item-expression for item in iterator if conditional].

Let’s look at some examples to see how they work:

x = [i for i in range(10)]

Produces the result:

[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

And utilizing a conditional statement:

x = [i for i in range(10) if i > 5]

Returns the list:

[6, 7, 8, 9]

because the condition i > 5 is checked.

As well as being more concise and readable than their for-loop equivalents, list comprehensions are also notably faster.

Nested list comprehensions

Generating, transposing, and flattening lists of lists becomes much easier with nested list comprehensions. Most of the keywords and elements are similar to basic list comprehensions, just used again to go another level deeper.

To demonstrate, consider the following example:

[(x, y) for x in [1,2,3] for y in [3,1,4] if x != y]

The result is:

[(1, 3), (1, 4), (2, 3), (2, 1), (2, 4), (3, 1), (3, 4)]

Complex expressions and nested functions

You can also use functions and complex expressions inside list comprehensions.

For example:

from math import pi  # import pi number from math module

[str(round(pi, i)) for i in range(1, 6)]

Returns:

['3.1', '3.14', '3.142', '3.1416', '3.14159']

List comprehensions are ideal for producing more compact lines of code. They can also be used to completely replace for-loops, as well as map(), filter(), and reduce () functions, which are often used alongside lambda functions.

Dictionary comprehensions

In Python, dictionary comprehensions are very similar to list comprehensions – only for dictionaries. They provide an elegant method of creating a dictionary from an iterable or transforming one dictionary into another.

The syntax is similar to that used for list comprehension, namely {key: item-expression for item in iterator}, but note the inclusion of the expression pair (key:value).

Let’s look at a simple example to make a dictionary. The code can be written as

dict([(i, i+10) for i in range(4)])

which is equivalent to:

{i : i+10 for i in range(4)}

In both cases, the return value is:

{0: 10, 1: 11, 2: 12, 3: 13}

This basic syntax can also be followed by additional for or if clauses: {key: item-expression for item in iterator if conditional}.

Using an if statement allows you to filter out values to create your new dictionary.

For example:

{i : i+10 for i in range(10) if i > 5}

Returns:

{6: 16, 7: 17, 8: 18, 9: 19}

Similar to list comprehensions, dictionary comprehensions are also a powerful alternative to for-loops and lambda functions. For-loops, and nested for-loops in particular, can become complicated and confusing. Dictionary comprehensions offer a more compact way of writing the same code, making it easier to read and understand.

Nested dictionary comprehension

In Python, dictionary comprehensions can also be nested to create one dictionary comprehension inside another.

For example:

{(k, v): k+v for k in range(2) for v in range(2)}

Returns:

{(0, 0): 0, (0, 1): 1, (1, 0): 1, (1, 1): 2}

Take care when using nested dictionary comprehensions with complicated dictionary structures. In such cases, dictionary comprehensions also become more complicated and can negate the benefit of trying to produce concise, understandable code.

Generator expressions

Generator expressions are yet another example of a high-performance way of writing code more efficiently than traditional class-based iterators. To better understand generator expressions, let’s first look at what generators are and how they work.

Generator functions

Class-based iterators in Python are often verbose and require a lot of overhead. Generators, on the other hand, are able to perform the same function while automatically reducing the overhead.

Generators are relatively easy to create; a normal function is defined with a yield statement, rather than a return statement. The yield statement has the effect of pausing the function and saving its local state, so that successive calls continue from where it left off.

Generators work in the following way:

• When a generator function is called, it does not execute immediately but returns a generator object.

• The code will not execute until next() is called on the generator object.

• Once yield is invoked, control is temporarily passed back to the caller and the function is paused.

• Local variables and their execution state are stored between calls.

• StopIteration is raised automatically when the function is complete.

Generator functions can be written as:

def gen():

   for x in range(10):

       yield x**2

Generator expressions

Generator expressions make it easy to build generators on the fly, without using the yield keyword, and are even more concise than generator functions.

The same code as the on in the example above can be written as:

g = (x**2 for x in range(10))

Another valuable feature of generators is their capability of filtering elements out with conditions.

Let’s look at an example:

gen_exp = (i ** 2 for i in range(10) if i > 5)

for i in gen_exp:

  print(i)

Returns:

36

49

64

81

Generator expressions

The syntax of generator expressions is strikingly similar to that of list comprehensions, the only difference is the use of round parentheses as opposed to square brackets.

For example, a generator expression can be written as:

(i ** 2 for i in range(10) if i > 5)

Compare that to a list comprehension, which is written as:

[i ** 2 for i in range(10) if i > 5]

Where they differ, however, is in the type of data returned. While a list comprehension will return the entire list, a generator expression will return a generator object. Although values are the same as those in the list, they are accessed one at a time by using the next() function.

In terms of speed, list comprehensions are usually faster than generator expressions, although not in cases where the size of the data being processed is larger than the available memory. On top for that, because generator expressions only produce values on demand, as opposed to list comprehensions, which require memory for production of the entire list, generator expressions are far more memory-efficient.

Generator expressions are perfect for working large data sets, when you don’t need all of the results at once or want to avoid allocating memory to all the results that will be produced. They are also perfect for representing infinite streams of data because only one item is produced at a time, removing the problem of being unable to store an infinite stream in memory.

As with list comprehensions, you should be wary of using nested expressions that are complex to the point that they become difficult to read and understand.

Wrapping Up

Python for-loops are highly valuable in dealing with repetitive programming tasks, however, there are other that can let you achieve the same result more efficiently. List comprehensions and dictionary comprehensions are a powerful substitute to for-loops and also lambda functions. Not only do list and dictionary comprehensions make code more concise and easier to read, they are also faster than traditional for-loops. The key to success, however, is not to let them get so complex that they negate the benefits of using them in the first place.

Similarly, generators and generator expressions offer a high-performance and simple way of creating iterators. Although similar to list comprehensions in their syntax, generator expressions return values only when asked for, as opposed to a whole list in the former case. As a result, they use less memory and by dint of that are more efficient.