I don't know if I'm missing something or it's a bug but I've set the setting "Allow non members to comment" to "no" but guest can still comment. On my blog it says "Please login first in order for you to submit comments" but the comment box is still there and allows anyone to post.
http://hullarity.co.uk/news/entry/welcome-to-hullarity-comedy-club