Category View and showing only selective modules

I have certain modules on the Home page, and would like to have other modules displayed on a Category view page that I have linked to a menu item. However, it's messing with which modules display. Basically I have a Banners extension that displays in a module on the home page, and I want a different module to only display on the category page. It isnt working as I was hoping, all modules are displaying on both Home and the category view. I hope this makes sense!

Thanks