Working Notes: a commonplace notebook for recording & exploring ideas.
Home. Site Map. Subscribe. More at expLog. — Kunal

Notes on PyTorch

torch.distributed

• this has a lot of magic functions available in the torch.distributed package

  • imported from distributed_c10.py link

• using it in a notebook as a way to use all 8 gpus on demand?

  • use the jupyter kernel as a torchrun controller equivalent
  • spawn multiple copies of the kernel remotely or locally
  • ideally this is baked into the kernel as a package instead, tentatively the way ray works?

• pytorch overrides the system excepthook... to prefix the exception with the rank number

• the magic WORLD property on groups comes from a metaclass

  • introduced in this Pull Request

• notes/pytorch
• notes/pytorch_torchscript
• notes/ml/torch_dispatch
• notes/python/descriptors
• notes/amazon_sagemaker_debugger
• notes/mathpy
• notes/gpumode/001_profilecuda
• notes/neural_nets
• notes/python/metaclass
• notes/python/typing_cheatsheet