In the second part of this series on deep diving into asyncio and async/await in Python, we will be looking at the following topics:

• task, task groups, task cancellation
• async queues
• async locks and semaphores
• async context managers
• async error handling

For as long as I have worked in Python land, I never had to touch the async part of the language. I know that asyncio library has gotten a lot of love in the past few years. Recently I’ve came across an opportunity to do a lot of IO and non-cpu bound work in Python. I decided to take a deep dive into the asyncio library and see what it has to offer.

In part 1 of this series (I originally just wanted to write one post and realized the scope is way too big), we’ll cover:

• How async code interfaces with synchronous code in Python
• How to convert synchronous code to asynchronous code, including how to prevent blocking of the event loop via custom ThreadPoolExecutor
• How to use asyncio to run multiple tasks concurrently

Basic example, async hello world

import asyncio

async def hello_world():
    asyncio.sleep(1)
    print("Hello world")

asyncio.run(hello_world())

>>> Hello world

Running two async functions in parallel

import asyncio

async def foo():
    while True:
        asyncio.sleep(1)
        print("foo")

async def bar():
    while True:
        asyncio.sleep(1)
        print("bar")

asyncio.run(asyncio.gather(foo(), bar()))

What if I have existing synchronous methods?

We can wrap a synchronous function in an async function, an example implementation would be a decorator (i love decorators, btw):

def async_wrap(
    loop: Optional[asyncio.BaseEventLoop] = None, executor: Optional[Executor] = None
) -> Callable:
    def _async_wrap(func: Callable) -> Callable:
        @wraps(func)
        async def run(*args, loop=loop, executor=executor, **kwargs):
            if loop is None:
                loop = asyncio.get_event_loop()
            pfunc = partial(func, *args, **kwargs)
            return await loop.run_in_executor(executor, pfunc)

        return run

    return _async_wrap

The above decorator is a higher order decorator (it takes arguments and then generates another decorator), example usage is the following:

import asyncio
import time

@async_wrap()
def foo():
    while True:
        time.sleep(1)
        print("foo from sync")

async def bar():
    while True:
        asyncio.sleep(1)
        print("bar from async")

asyncio.run(asyncio.gather(foo(), bar()))

What is copiable anyway?

Python is garbage collected and has a reference counting system. This means that when you create an object, it is stored in memory and a reference to it is stored in a variable. When you assign a variable to another variable, the reference count for the object is incremented. When you delete a variable, the reference count is decremented. When the reference count reaches zero, the object is deleted from memory.

This is a very simple explanation of how Python works. There are many more details that I will not go into here. The point is that when you assign a variable to another variable, you are not creating a copy of the object. You are creating a new reference to the same object. This is important to understand because it can lead to some unexpected behavior.

Questions I had:

• What happens when you assign a variable to another variable?
• What happens when you return a complex object (i.e. a class) as part of a tuple from a function?
• What happens when you spin up a subprocess, call a method you defined in one class, and give it an object as an argument?

Since Python type hints are introduced, they have made complex Python code-bases much more readable and easier to maintain - especially combined with newer static analysis tools such as mypy or pylint. However, even with these tools, Python is still a dynamic language. When using a dynamic language on a larger application (>5k LOC), the ability to do whatever we wanted any where and any time is more of a curse than a blessing.

In this post, I wanted to discuss several options of implementing loosely coupled code in large Python codebases that I have played around with and the final solution of dependency injection based pattern I ended up deciding on.

A collection of weird things in Javascript:

1. var scoping rules

for (var i = 0; i < 3; ++i)
{
	const log = () => {
  	console.log(`a ${i}`);
  }
  setTimeout(log, 100);
}

for (let i = 0; i < 3; ++i)
{
	const log = () => {
  	console.log(`b ${i}`);
  }
  setTimeout(log, 100);
}

The output here is:

"a 3"
"a 3"
"a 3"
"b 0"
"b 1"
"b 2"

Why does var cause it to print 3?

2. const in Javascript does not mean the same as C/C++. Example:

const value = 3;
value = 4; // error, cannot override a constant
value += 3; // error

const obj = {a : 3};
obj.a += 3; //allowed
obj.a = 5; //allowed

Turns out const in Javascript is more of a “const” reference like const & in C++. It does not mean the value itself is constant - just the reference to the array cannot be changed.

3. Converting time formats can be tricky

Suppose you have a time in yyyy-mm-DD format and you want it in mm/DD/yyyy format.

new Date('2016-06-05').
  toLocaleString('en-us', {year: 'numeric', month: '2-digit', day: '2-digit'})

// Output:
>>> '06/04/2016'

Wait, what happened?, I asked for 2016-06-05 in mm/dd/YYYY but it gave me 06/04/2016 instead! This because all dates by default assumes it’s GMT time, when you convert it to a local timezone, you might get a different date.

The moment library fortunately makes this a lot easier.

var date = new Date('10/01/2021');
var formattedDate = moment(date).format('YYYY-MM-DD');

If we don’t want some extra dependency, it’s probably easier to just not convert the date into a Javascript Date obj and directly do string operations on it to get it to the format you want. Example:

function reformatDateString(dateString) {
    //reformat date string to from YYYY-MM-DD to MM/DD/YYYY
    if (dateString && dateString.indexOf('-') > -1) {
        const dateParts = dateString.split('-');
        return `${dateParts[1]}/${dateParts[2]}/${dateParts[0]}`;
    }
    return dateString;
}

While browsing the excellent modern C++ reactive console UI library FTXUI, I noticed this piece of code in how the events are typed/handled.

// event.hpp
struct Event {
  // --- Constructor section ---------------------------------------------------
  static Event Character(char);
  static Event CursorReporting(std::string, int x, int y);
  // Other constructor methods

  // --- Arrow ---
  static const Event ArrowLeft;
  static const Event ArrowRight;
  static const Event ArrowUp;
  static const Event ArrowDown;
  // .... Other definitions etc....

My immediate reaction to this is that this feels weird - partially because I am not used to the author’s C++ style, in addition, the combination of static member variables sharing the parent-type, and static methods for constructors are really confusing to read.

Let’s dive into it to see how things work.

Internet consensus tends to label C++ as a hard language; I like to think Cpp is a “deep” language. There are always rooms for improvement - doesn’t matter how long you have been coding in C++. The expressiveness and the depth of the language is double-edged. It is what makes C++ great, but also makes it daunting for new users. These are the mistakes I’ve made in my daily usage of C++. I hope they can be useful for other people to avoid them in the future.

1. Capture by reference on transient objects

Callbacks (lambda functions, function pointers, functors, or std::bind on static functions) are a common paradigm when you work with message queues, thread pools, or event based systems. Lambda and closures give you a lot of power - but too much power could often cause problems, consider the following code:

This is me trying to reproduce the results in this Stack Overflow post and Dan Cole’s blog.

Things to keep in mind:

• Include morley.csv (Search local JavaScript for /morley.csv)
• Include D3.js
• Include box.js
• Include the local CSS and JavaScript

Check out the code for this post on GitHub.

Hacker News recently had a post about “maximizing luck surface area”. Jason Roberts made a point that seemingly random success that people experience that most people attributes to luck, is really a game of probability.

While most people follow the axiom of doing “good work” by spending years going through school, honing their craft, and then practicing those crafts and skills to perfection at work (cough, programming languages, frameworks, design patterns, algorithms, etc.) Many people neglects the other important pre-requisites to success - letting others know about your work and building up the reputation for network effects.