A frame-based 4-score disease severity prediction architecture is proposed with the integration of deep convolutional and recurrent neural networks to consider both spatial and temporal features of the LUS frames and is found to be very effective in detecting COVID-19 severity scores from LUS images.