If you have a "Contact" (not account) attached to an Account, and need to remove it. You can't remove the Account Contact Relation if that Contact is a Direct Member. You need to edit the contact, and change it's account parent to a new Account. Then you can remove the relation to the old Account.