hey jasper,
unfortunately that's not possible currently - only the other way round: followers are optional but friends not.

but we have a whole area dedicated to your ideas and feature requests at http://stackideas.com/voices
feel free to post your idea over there instead of the forum as that's the place we're monitoring when it comes to planning for feature implementation in upcoming versions.

have a nice day