As far as I'm aware, it's all done in real-time, so your page doesn't refresh when you're in the middle of watching a video. YouTube haven't done it very well so there's a lot of room for error.
It happens to me every now and then too. It tends to work most of the time when you press a link that is within the YouTube domain. Odd.
