A differentiable penalty is proposed, which enforces inequality constraints directly in the loss function, avoiding expensive Lagrangian dual iterates and proposal generation and has the potential to close the gap between weakly and fully supervised learning in semantic medical image segmentation.