Richard,
Your server could be delaying slightly in the case of non-existent domains. When this happens ListMail tries to 'skip' the message. On servers which are just a bit too slow to respond the next SMTP response is not what is expected. Try adding the following line to your config.php to allow for a larger timeout when sending messages.
$smtp_timeout = 30;
Regards