A Hierarchical Relational Network that computes relational representations of people, given graph structures describing potential interactions, is presented that learns relational feature representations that can effectively discriminate person and group activity classes.