I have deleted group categories on both my production and dev domains and in both cases when deleting a group category from the back end it will produce error with primary key.
Thanks for the access. I've tried to replicate the issue on your site by create and delete the group. The group is successfully deleted: http://screencast.com/t/IFnkdKphm . Did I missed anything?
It does not matter the name, create any category and then try to delete it and you will see the error. I have created this video to assist you.... https://www.youtube.com/watch?v=pNNjQWTC4js
If you want to replicate the error do it more than once. I put video in this thread so you can see and I just tested again and first time was ok, then the second time it gave duplicate categories and then gave the error I posted.