FP in python in the real world

Javier Rojas

March, 2021

Map for this talk

No Shared (Mutable) State
Functions as first-class citizens
Closures

No Shared (Mutable) State

Immutability
Pure Functions
Referential Transparency

Sorry

Functions can change:
- their arguments
- global variables
Python doesn’t enforce immutability

Why be sorry?

Pure functions make code easier to read

With immutability:

More compiler optimizations are available
Sharing data is easier and safer
Concurrency is MUCH easier to get right

Sorry Not Sorry 😅!

On the other hand, things like immutability are useful, specially in complex environments with high concurrency, yes. But, they are rather cumbersome, in day-to-day simple code.

for example, the code below shows the declaration of a dictionary in Elixir, which is similar to what you see in most programming. But, if you want to update an element of the dictionary, which is what line 2 does here, you can’t just do that; you have to recreate the dictionary to make the change.

and it gets worse with nested data structures; line 4 shows a dictionary of dictionaries, which you can also declare in a natural way.

but, if you want to update a field of one of the dictionaries inside, instead of doing the update, like what you have in line 6, you have to re-create the internal dictionary with the update you want, and then recreate the whole dictionary, with the updated “internal” dictionary.

And this is not only cumbersome to read and to write, it’s also heavy on memory consumption, since all these objects are being created and destroyed continuously. It’s not the end of the world, and there are helpers for this, but it’s important to keep in mind that this is what is happening under the hood.


d = %{age: 7, height: 110}
d = %{d | age: 9}

d = %{john: %{age: 7}, meg: %{age: 20}}

# d[:john][:age] = 8
d = %{ d | john: %{ d[:john] | age: 8 } }

Functions as first-class citizens

Functions as variables

Functions as arguments to functions

Functions as return values

Dynamic defaults


from uuid import uuid4

def prefixed_uuid():
    return "acc-" + uuid4().hex

class Account(models.Model):
    ...
    public_id = StringField(default=prefixed_uuid)

Decorators

Another really common example of using functions as variables in Python is creating and using decorators, which are a way of adding a pre-processing or post-processing step to any function.

For example, let’s consider a function that implements an endpoint in a web application, like what we see on lines 10 and 11, and suppose we have to add authentication to it: before running the code for the endpoint, we have to validate some of its parameters, if the check fails send the user somewhere else, and if it succeeds run the code for the endpoint.

And we want to do this in a reusable way, because we have to use this check in many endpoints, not just on this one; in Python, a natural way to do this is to use a decorator, and use it to decorate all the endpoints you need.

A decorator is just a function that receives the function we want to wrap, or decorate, and gives you back a new function, that usually does some work before, or after, the original function; in this example it performs the authentication check.

And decorators not only use functions as return values, but also as variables; when you apply a decorator to a function, as in line 9, you are just replacing the original function with your new one, and while you can use the syntax with the at symbol, what Python does is just update a variable, as in line 13


def authentication(view_func):
    def authenticated_view(*args, **kwargs):
        if args[0] == 'valid_user':
            return view_func(*args, **kwargs)
        return redirect_to_login()

    return authenticated_view

@authentication
def welcome_view(username):
    print(f"hello {username}!")

welcome_view = authentication(welcome_view)

Higher-order functions: map, filter

Another place where you see the idea of a function as a first class citizen is in Python’s map function, which takes a function, and a list, and calls that function once for each element in the list, and gives you back another list with the results, like the example here, that calculates the squares of a list of numbers.

However, Python has another way of writing these kinds of instructions: the list comprehension, which does the same thing, perhaps in a more verbose way, but perhaps more readable, in my opinion.

Python also has the classical filter function, which given a function and a list, will give you another list only with the elements for which the filter function returns a boolean True. This is another common task, and is a task that can also be expressed with list comprehensions, again, perhaps a bit verbose, but more readable.

Map: apply a function to all the elements of a list:


map(lambda x: x * x, range(5))
[0, 1, 4, 9, 16]


[x * x for x in range(5)]
[0, 1, 4, 9, 16]

Filter: Keep only some elements from a list:


filter(
  lambda x: x % 2 == 0,
  range(5)
)
[0, 2, 4]


[
  elem
  for elem in range(5)
  if elem % 2 == 0
]
[0, 2, 4]

Other higher-order functions

itertools
- takewhile, dropwhile, groupby, …
functools
- reduce, cache (memoize), partial (currying), …

Closures


def external_service():
    """Fetches a number from a remote endpoint"""
    return 41

def action(retries=3):
    """"Obtains number from a service, adds 1 to it.
    Retries in case of timeout
    """
    try:
        v = external_service()
    except Timeout as exc:
        if retries > 0:
            return action(retries=retries - 1)
        else:
            return 0

    return v + 1

Closures

We have to test that, if external_service times out, the whole action gets retried.

How can we do that? the approach I’ll follow here is: I’m gonna mock the external service, and replace it with a function that raises timeout the first time I call it, and succeeds the second time.

Now the only difficulty is: how can I make my mock function remember how many times it’s been called, and act according to that? one quick way of doing that is using a closure

(switch to examples/closures.py)

here we see the code we want to test, and a small test function for it, which implements the plan: replace external service with a mock, call the code to test, and check that we get the expected result.

now let’s look at the mock, which is what I described: a piece of code that checks and updates how many times the code has been called, and depending on that it raises timeout, or continues with the original code

the important thing to note here is how the code uses the times_called variable; it’s a variable that is defined outside of the mock function, but nonetheless the mock can access and update it. It’s almost like a global variable, except that it’s created as part of the test function, and it can be created many times; if we for example call the test function several times, each time we will get a new variable.

Also, note here that Python forces us to tell it where this variable came from: it’s not an argument, it’s not a variable in this function, it’s not a global variable, but it’s a variable from the enclosing environment; we express this in Python with the nonlocal keyword.

We can run the test, and see that it works properly: the code to be tested is exercised, the mock intercepts the call to the service, times out, and when called again runs the original service as normal, and we can validate that the mock was actually called twice.

Goal: test retries in action

how?

Mock external_service to timeout once and succeed afterwards

But, how to make the mock remember how many times it’s been called?

using a closure

(see closures.py)

What do we get from closures?

private reference to variables

reference and update that private state

from some component created dynamically

Classes + callable objects

With Python, we can, thanks to an interesting feature that can be added to Python classes: if a Python object has a method called call, you can use the object, and call it as if it was a function, and Python will run the code in the call method for you under the hood.

(switch to examples/closures.py)

here we see the test function, which is fundamentally identical to the one we had before, the only thing that changes is how we initialize the mock; previously we handled the state explicitly, now we just create an instance of a class.

and let’s look at the class: it has a constructor, where we set-up the initial state for the amount of times it has been called, and we store the function we mock up, to call it later.

and then there is the call method, which does what the mock is supposed to do: check how many times it’s been called, and act accordingly, and which thanks to Python can just act as a function without any other changes to the code that will use the mock.

A class can define a __call__ method, which will be called when instances are “called” as functions.

(see closures.py)

Things not discussed

types
pattern matching
macros

FP in python in the real world

Map for this talk

Map for this talk

No Shared (Mutable) State

No Shared (Mutable) State

Sorry

Why be sorry?

Sorry Not Sorry 😅!

Functions as first-class citizens

Functions as first-class citizens

Dynamic defaults

Decorators

Higher-order functions: map, filter

Other higher-order functions

Closures

Closures

Closures

What do we get from closures?

Classes + callable objects

Things not discussed

Things not discussed

Thanks!