Scalability and Distributed Training

How data parallelism scales model training across devices.

In data-parallel distributed training, what is split across the devices?