Nope, not always.
In some cases you could deduct that someone has removed you. But that is only in certain cases (like you just explained). The problem with that is that you don't know up front what the case is.
In your example, you knew that you've removed that other account and thus you can correctly determine that one account has removed the other in its list.
But in practice, you don't know if you're friend has realy removed you from his list, or if it is another scenario. In both cases you'll get the same "you have invited" or similar note.
And because of that, it is safe to say that you can't know for sure if someone has delete you. It is always a guess and thus never accurate.
I hope this confusing explanation makes some sense though
/ 
