Python Generator DSLs

Last modified on November 2, 2020 by Alex


Python generators have a curious ability to both to yield values from generators (suspending computation), and receive values from a yield, which is far less known. This is equivalent to Kotlin’s coroutines and allows using them for DSLs.

Python Generators

If you are reading this, you presumably know about generator syntax in Python:

What is less known is how to “drive” a generator manually:

That generators can return values:

And that generators can actually accept values as “results” of yields:

In fact, we can simplify our loop a bit since the initial __next__() call can be replaced with send(None),

Monadic DSLs

Using a combination of an ad-hoc algebraic data type and generator constructs, we can make a simple monadic DSL:

Now we need some way to run it:

We can separate execution of individual action from running an entire computation:

Inversion of inverted control

Often you encounter APIs that have so-called “inversion of control” - you are not controlling the main loop of the application, some other piece of code is, and you are supposed to handle “incoming events” using a fixed set of callbacks:

This is a fairly common but annoying design. Some of the issues with this design:

  1. All communication between callbacks has to be done through a mutable state defined on the Handler.
  2. Transitions between fundamentally different logical states have to be handled through some sort of hierarchy of state machines and handlers delegating messages down that hierarchy.
  3. Logic is spread out across multiple different callbacks. The higher the granularity of the callbacks, the more spread-out it is.

Can we “re-invert” control flow in such a situation?

Indeed we can, with the use of monadic generator DSLs: