I think it's modifying the vmx file, manually adding something like mks.enable3d and modifying the video memory size.
You can do that with a linux host because it's pretty much still software enumeration anyway if I get it right.
It's been experimental since VMWare 5.5 whereas I remember 6.0 did not add anything new to this; only dx8 is supported.
Ok, found it:
http://www.vmware.com/support/ws55/doc/ws_vidsound_d3d.html