A novel deep learning architecture to address the abovementioned difficulties and also an efficient training framework for set-to-set matching is proposed and evaluated through experiments based on two industrial applications: fashion set recommendation and group re-identification.