It sounds that some file may have got corrupt. You can try the following procedure to remove all Custom Emoticons related things. This will remove all emoticons you have installed.
Go to the folder
C:\Documents and Settings\Username\Application Data\Microsoft\MSN Messenger\number, with Username being your Windows username and number the number that symbolises your email address. Go to
this page to find out which number that is.
Open that numbered folder and delete the folder called CustomEmoticons. Reboot and add your custom emoticons again; your problem should be fixed
/ 
