This paper proposes a simple but effective criterion called NovelD, which solves all the static procedurally-generated tasks in Mini-Grid with just 120 M environment steps, without any curriculum learning and finds that empirically it helps the agent explore the environment more uniformly with a focus on exploring beyond the boundary.