Spatial Decomposition Network (SDNet) is proposed, which factorises 2D medical images into spatial anatomical factors and non-spatial modality factors and is ideally suited for several medical image analysis tasks, such as semi-supervised segmentation, multi-task segmentation and regression, and image-to-image synthesis.