if you want to remove them from your allowed list, click on their name then click the 'Block >>' button
also the option being greyed out is not because you are blocked or unblocked... that information is not sent to you... its because they are still on your contact list (from what my quick testing told me)
/ 
