This work proposes virtual point tracking for real-time target-less dynamic displacement measurement, incorporating deep learning techniques and domain knowledge and demonstrates the approach for a railway application, where the lateral displacement of the wheel on the rail is measured during operation.