It seems that someone experienced is using tactics to make you look bad. The spammers are very likely NOT sending from your web site. What is happening could be that they are setting the "From" address for their mailings to an address on your domain. The "From" address, used for display and replies can, on most servers, be set to anything the user wants regardless if the address exists or not on that server. This can cause all sorts of problems like some hosts mistakenly flagging you as the sender and/or bouncing email back to you. On some servers, senders can even set the "bounce" address to anything they want, which makes it even harder to avoid confusion.
If you look closely at the headers of the messages you will almost certainly find that the originating server is not your own. You may see your domain in the "Return-path" or the "From" header, but this does not necessarily mean the message(s) originated from your server. Look for the first "Received from" header for this.
Unfortunately, there isn't much you can do to stop other people forging your address and, again unfortunately, you will have to deal with any bounces that come to you by deleting them. IMHO, blacklisting is practically inevitable these days anyway, even when running an opt-in list. You will have to rely on the response of your subscribers who do not use blacklists or filters.
If you see people have signed up to your autoresponder, this is a bit different. Basically, anyone can freely subscribe to your newsletters. If you notice it's always from the same, or similar, IP address, we could employ a manual block of anyone subscribing from the IP by modifying your signup.php file. The next version of ListMail will feature a ban address (and, I just realized the need for) banned IP address support.