It does appear that if your messages were not set they are lost
I'd rather come to the same conclusion myself.
3G for a mailout? That seems high... are you sending attachments?
Yes, the mailout is html with a few inline images. I wanted to run some size reduction but didn't quite have the time.
I must recommend a link to your file instead as there always seem to be undesired results with attachments to large lists.
We looked at that, and while lower in initial bandwidth (and probably overall as addresses bounce and recipients don't load images) we don't like the "click to load remote images" that you get with most email clients - it just doesn't look professional. Remote images always seem to me to smack of recipient tracking. I hate spam as much as anyone, and tracking always gets my goat.
I suppose what must have happened is ListMail erroneously "skipped past" a number of your remaining emails, most likely due to the smtp timeout issue.
Seems that way - yes, the SMTP diagnostic would be good (drat, drat and double drat; curse my metal body, I was too slow) but I'm wondering what LMP could have done to record its own action after such a failure mode... silently losing the remainder of a mailout is, mmm, not ideal.
It sounds like you have a plan now that is much easier than rebuilding the lm_send* tables manually...
It seems to have gone pretty well. We got a sensible looking "remainder to send" list, and LMP has not had any problems sending it - it finished a few minutes ago.
Now comes the problem that the queue seems to be being processed very slowly. Time to kick off a bunch of queue runners in parallel, I think.
Another couple of ideas came to mind:
pre-loading DNS - with a local BIND running and a sizeable cache, one could perhaps make a script that would look up MX records for the domains in the user list, causing them to be cached locally; then when sendmail comes to look them up, they'll already be here. You could also heavily parallelize that, unlike the serial processing of sendmail (smtp or single queue running).
progress monitoring - I wrote a SQL fragment that can pull out the status of the queue and store it in another table, where it can be queried to produce graphs both realtime and after completion of the rate of sending. This is quick and dirty, but works:
CREATE TABLE jmb_stats_send (
id SMALLINT(5) UNSIGNED, -- per lm_sent.id = mailout identifier
date DATETIME, -- really, we should have a unique key across id/date
total MEDIUMINT(8) UNSIGNED,
remaining MEDIUMINT(8) UNSIGNED
);
INSERT INTO jmb_stats_send
SELECT s.id,
NOW() AS date,
s.numsent AS total,
count(q.id) AS remaining
FROM lm_sendq q, lm_sent s
WHERE q.mid = s.id
GROUP by s.id;
SELECT * FROM jmb_stats_send;I stuck the INSERT into a small php and loaded it via cron like the resume script.
