It's just the way Windows Live Messenger works. It is designed to specifically open a
new Internet Explorer window no matter what your defaults are. As you probably remember, WLM 8 was released when IE6 was around (which didn't have tabs) so it didn't matter back then.
You can probably bypass this issue if you use StuffPlug but there is no version for 8.0 or 8.1 yet. Check out
www.stuffplug.com and see if it has been released.