GPU memory is the most constrained resource in deep learning. When you scale from a single GPU to distributed training using DistributedDataParallel (DDP) or Fully Sharded Data Parallel (FSDP), me…