Working Notes: a commonplace notebook for recording & exploring ideas.
Home. Site Map. Subscribe. More at expLog. — Kunal

Asyncio

Developing with asyncio https://docs.python.org/3/library/asyncio-dev.html
PYTHONASYNCIODEBUG = 1
=-X dev=
=asyncio.run(debug=True)=
=loop.set_debug()=
Enable =ResourceWarning=
logging.getLogger("asyncio").setLevel(...)
AOSA Book: Web crawler with Asyncio Coroutines [2020-05-09 Sat]

http://www.aosabook.org/en/500L/a-web-crawler-with-asyncio-coroutines.html

TODO Understand how non-blocking sockets actually work
TODO Dig into the different event loops
Understand how the different event loops sockets actually work at the kernel / C level, and visualize it using ebpf.
TODO Check out the selectors module in python
TODO Read through the new asyncio modules, read about =Event=
Notes
"Stack Ripping": Loss of context because of the broken event loop.
Coroutine takes ~3k of memory, compared to ~50k per thread
"Note a nice feature of async programming with callbacks: we need no mutex around changes to shared data"
PyEval_EvalFrameEx takes a frame object and evaluates bytecode in the context of that frame.
inspect module also looks pretty good for looking at where I am
Python stack frames are allocated in the heap!
How generators work:
- gen_fn.code.co_flags & generator_bit = 1
- when python "compiles" the gen_fn to bytecode
- creates generators when the flag is set
- Python generator = stack_frame + code
  - =gen.gi_code.co_name=
  - The stack frame isn't on any actual stack
  - Frame has a last instruction pointer: =f_lasti=
Coroutines
- Run with a driver that waits for futures to resolve
- As part of the callback for the resolution, sends out the next step
- Using "send" from generators
- The event loop is a distinct concept
This would be much harder to read if I hadn't extensively used coroutines in Advent of Code.
Python's =yield from=
- Makes a generator act as if it were another generator
- The return value of the inner generator is the value sent to the outer generator
Handles exceptions correctly as well!
To callers, advancing a generator is the same as advancing an iterator
"The real framework addresses zero-copy I/O, fair scheduling, exception handling, etc"
"Real" AsyncIO
- =from asyncio import Queue=
  - =yield from q.get()=
  - =q.task_done()=
  - =q.put_nowait()=
- =loop = asyncio.get_event_loop()=
- =loop.run_until_complete(...)=
- ClientSession does connection pooling, keep alives
- asyncio.Task <- actually drives the future forward
- Explicitly call worker.cancel() to clean up pending tasks
Generator cancel:
- Throw an exception into the generator from the outside
- Task explicitly sends a CancelledError to the coroutine
Coroutines are only interruptible at yield points
The crawler deals with redirects by stepping through them to avoid redownloading the same pages again and again.
A Task is a Future that /resolves itself/
yield from --> =await=
Asyncio & Music
- TODO Check out UVLoop Understand how nodejs's event loop works.
- TODO Write about /how/ to digest a technical document/book
- TODO Write about /why/ to digest a technical document/book