It's far more likely that it's a bug in how you use it than it being related to x64.
Messenger is a x86 (32-bit) program, and as such is run in a x86 enviroment with x86 APIs. x86 processes do not have access to x64 APIs, and as such EM_SETSEL would just take 32-bit parameters.
The article you mentioned is about 16-bit vs 32-bit, not about 32-bit to 64-bit, and on top of that it checks on-compile time, meaning that if you'd compile your app as 16-bit, it'd have used the 16-bit methods. Messenger is compiled as 32-bit, and as such should just use 32-bit APIs.
Furthermore, changing your programs behaviour on x64 windows even though you're a x86 process would only introduce weird bugs.
I'd suggest you read up on EM_SETSEL
here and try to find out where you went wrong.
Also I suggest you test your script on a 32-bit machine, and I can almost assure you that the same bug will appear, as it's most definitely not related to your 64-bit OS