Code QuizAdvanced

DDP Gradient Synchronization

Spot why gradients are not synced across processes in this DDP setup.

Codepython

import torch
from torch.nn.parallel import DistributedDataParallel as DDP

def build(rank):
    model = MyModel().to(rank)
    # ready for distributed data-parallel training
    optimizer = torch.optim.SGD(model.parameters(), lr=0.01)
    return model, optimizer

Why won't gradients be synchronized across GPUs in this code?