Silent by design: why a missing await on run_in_executor survives review

A background service ran fine for weeks. Every so often one processing cycle would act on data that was incomplete— a few records simply missing. No exception. No warning. Nothing in the logs. Restarting made it go away, so for a long time it was filed under “probably the network”.

The cause was two lines that look completely ordinary, and a language behaviour that most Python developers assume works the other way around.

1. Fifteen lines that reproduce it

This is not the production code — it is the smallest thing I could write that fails the same way. Run it yourself; that is the point.

import asyncio, time
from concurrent.futures import ThreadPoolExecutor

state: dict[str, str] = {}

def load(key: str) -> None:          # stands in for a DB query
    time.sleep(0.05)
    state[key] = "loaded"

async def build_state(loop, ex) -> dict[str, str]:
    loop.run_in_executor(ex, load, "agents")      # <- no await
    loop.run_in_executor(ex, load, "equipment")   # <- no await
    return dict(state)                            # returns while threads still run

async def main():
    ex = ThreadPoolExecutor(4)
    for i in range(5):
        print(f"run {i+1}: {await build_state(asyncio.get_running_loop(), ex)}")
        await asyncio.sleep(0.06)

asyncio.run(main())

Output:

run 1: {}
run 2: {'agents': 'loaded', 'equipment': 'loaded'}
run 3: {'agents': 'loaded', 'equipment': 'loaded'}
run 4: {'agents': 'loaded', 'equipment': 'loaded'}
run 5: {'agents': 'loaded', 'equipment': 'loaded'}

The first call returns empty. Every call after it looks correct. There is no error anywhere.

2. Why Python stays quiet

Most of us learned that forgetting await is loud — Python emits RuntimeWarning: coroutine was never awaited. That is true, and it is also the reason this bug is so easy to miss: the warning belongs to coroutines, and run_in_executor does not return one. It returns a Future, and Futures have no equivalent warning.

I checked this on Python 3.9 and on 3.14: identical behaviour. This is not a rough edge of an old release that has since been fixed.

One caveat worth stating precisely, because it is easy to overclaim here: if the function you hand to the executor raises, Python does eventually print Future exception was never retrieved. But it prints it when the Future is collected — not at the call site, not in the stack that caused it, and easily buried in a long-running server. And in the case that actually hurts, the function does not raise at all. It succeeds. It is just late.

3. Why nobody catches it in review

This is the part I find genuinely interesting. Look again at the output: the first call is wrong, and then the bug appears to heal itself.

It heals because state is a module-level global that survives between calls. By the second call the threads from the first call have finished and populated it. Call two is not reading its own data — it is reading leftovers from call one.

Run the same code with a fresh state object each time and the disguise falls away:

state GLOBAL (kept between calls):  [0, 2, 2, 2]   <- only the first call is wrong
state LOCAL  (reset every call):    [0, 0, 0, 0]   <- wrong every time

That is the whole reason this class of bug survives: you hit it once, reload, and it works. It does not reproduce on demand, so it never becomes a ticket. It waits for a cold start in production.

4. The obvious fix, and why it is not enough

Collect the futures and await them:

futures = [
    loop.run_in_executor(ex, load, "agents"),
    loop.run_in_executor(ex, load, "equipment"),
]
await asyncio.gather(*futures)
return dict(state)

This fixes the timing. It does not fix the second problem, and the second problem is worse.

If one loader raises, gather() re-raises — good. But the other loaders have already mutated the shared state. You are now left with a global that is half-updated, and it stays that way. The next request does not fail; it quietly reads a state that is part-new and part-old. A crash that leaves bad data behind is worse than a crash.

5. Build beside, then swap

The fix is not to add a backup-and-restore path around the mutation. It is to stop mutating the live object at all:

async def build_state(loop, ex) -> dict[str, str]:
    global state
    new_state: dict[str, str] = {}                  # build beside the live one

    def load_into(target, key):
        time.sleep(0.05)
        target[key] = "loaded"

    await asyncio.gather(
        loop.run_in_executor(ex, load_into, new_state, "agents"),
        loop.run_in_executor(ex, load_into, new_state, "equipment"),
    )

    state = new_state                               # swap only after all succeed
    return dict(state)

If anything fails, gather() raises before the assignment, and the previous state is still intact and still consistent. Readers never observe a half-built object, because the only thing that ever changes for them is one reference.

6. The general shape

None of this is new — it is copy-on-write, and it turns up everywhere once you recognise it. A git commit does not edit your previous commit. A blue-green deploy does not upgrade the running fleet in place. os.rename() is atomic precisely so that a writer can build a file beside the real one and then move it over.

The two ingredients that made the original bug are worth naming, because they travel together:

• Fire-and-forget concurrency — work started but never joined.
• Shared mutable state — so partial results outlive the request that produced them.

Either alone is survivable. Together they produce corruption with no error attached to it, and a symptom that disappears when you look at it.

7. How to check your own code

• Grep for run_in_executor and check every call site: is the returned Future stored, gathered, or awaited? A bare call on its own line is the smell.
• The same applies to asyncio.create_task() and ensure_future() — same pattern, same silence.
• Reset the shared state at the start of each call and run your tests. If results suddenly break every time instead of only on the first run, you have just made a hidden bug reproducible — that is the useful outcome, not a regression.

I found this while auditing an async data-loading path in a scheduling service. The report I wrote at the time said “add the missing await”. It took a second pass to notice that the missing await was the smaller half of the problem.