He's using API Hooking, using the Detours library. For example, it could change the IAT table to fake Messenger call CreateWindow, while it calls Patchous CreateWindow instead. There are sereval other methods, check CodeProject.com for samples. I recommend HookImportedFunction API.
He's also made his own RichEdit interface for the color codes etc.
It's also done by normal Windows Message hooks.
Oh, yeah, he added the button by intercepting LoadResource, FindResourceW, SizeofResource. Then he fakes Messenger to load another/modifed copy of the window resources with a button added.
Hope this helps
.