There's probably another method involving deleting a file in your storage map where also your other custom stuff is stored for Messenger, but that might remove all custom emoticons. Or, editing the proper file or something (decode, decrypting, editing, re-encrypting, re-encode, etc)..... but I would never bother doing that...
What I would do instead is using the "poor man's version
" of editing the standard emoticons by patching. This will probably work for this too...
Thus by patching ":}" and ":]" in msnmsgr.exe
to something else (eg: "11" and "22"), then changing the custom emoticons shortcuts to whatever in Messenger, and finally patching those build-in shortcuts back to what they were (or putting back the original msnmsgr.exe
Or as a step-by-step:
1) Close Messenger
2) Using a hex-editor you search for ":}" and ":]" (but as unicode text, thus not simply as ascii) in the file msnmsgr.exe
You will recognize the proper search results (there might be more than one) because there will be more emoticon shortcuts around the same place.
3) Note these addresses down.
4) Change those build-in shortcuts ":}" and ":]" to something like "11" and "22"
5) Save the file (obviously
6) Start Messenger and change your custom emoticon shortcuts to what you want.
7) Close Messenger
8) Open msnmsgr.exe
in your hex-editor again and go back to those addresses you found in point 3.
9) Change them back to what they were, thus ":}" and ":]".
10) Save the file
done (I hope
of course maybe the best way to do all this is by using a copy/backup of msnmsgr.exe in case you screw up.