Hopefully all users have written you from the address they subscribed with. If not, and if you include a remove link in your messages and they have quoted the message they received, you may be able to simply click the link to remove the user.
Regards