Model only learns when I train it for 10 epochs .

I am training a Siamese network to recognise if two digits are similar or different.

The model learns when I train for 10 epochs.

But when I train it for 20 epochs after reinitialising it, Its training and validation accuracy gets stuck at 50%. Also the loss doesn't reduce.

What might be causing it?

Note. When I train it for 10 epochs, the validation accuracy is always more than training accuracy.

Edit: Included the graph. Trained it only for 20 …

accuracy deeplearning digits loss network reduce train training validation