Distributed Learning of Deep Neural Networks using Independent Subnet Training

Independent subnet training is introduced: a simple, jointly model- parallel and data-parallel, approach to distributed training for fully connected, feed-forward neural networks, where it often results into boosting the testing accuracy, as it implicitly combines dropout and batch normalization regularizations during training.