Hmmzz, might need some listcleaning then.
Get
Noroom's ListManager
It could be your allowlist is too full, this program will help you clean all possible lists (f.e. people still in your allowlist that aren't on your list and who have deleted you from theirs).