Unexpected SMTP delay or failure with some domains

[mike2]

Hi Mike,

No, I'm not running sendmail, I'm running LMP own SMTP
mailer, from LMP Configuration page:

Code: [Select]

SMTP Server (recommended):
settings - port: 25, reconnect every 249 emails.

Let me know if you have any other ideas - this thing seems to be stalling constantly - about every 30-45 minutes!  

Is the 'reconnect every 249 emails' too low?  I used to have it at 499, but lowered it when I started having problems.

Right I understand that, but which SMTP software is running on the mail server to which your LMP connects.  In the same settings you are talking about where it says "Host", whatever machine that is has to be running an SMTP program that is listening on port 25 (IE>  Sendmail, qmail, postfix, etc).  

If you know how that is configured it may be helpful to diagnose the problem.  If it's something that just started maybe someone changed the config or something.  

This really sounds like a broken SMTP, but it's so hard to say since you say you weren't having this problem in V1.84, what about V1.85?  If so maybe Dean changed something in the SMTP routines that broke for your mail program.

Posting the SMTP logs would certainly help (not the whole thing, just where it stalls of course).

The best way for a SMTP program to be set up with LMP (IMHO) is basically for LMP to generate the message/headers, etc. and connect to your SMTP and say "Here's the message", your SMTP simply says "Ok, it's queued, Next Please" and on and on... At that point it is the SMTP programs job to deliver the message.  Most SMTP programs aren't necessarily set up Stock to do it this way, but can be configured to.

[BGSWebDesign]

Hi Mike,

This really sounds like a broken SMTP, but it's so hard to say since you say you weren't having this problem in V1.84, what about V1.85? If so maybe Dean changed something in the SMTP routines that broke for your mail program.

Funny thing happened - I moved the CRON up on resume.php to every 5 minutes - now it's not doing it at all - with still a large mailing - no stalling when it finished up last night, or on today's mailing?

Also, regarding v1.84 - I remember now I put my own hacks in to one of the modules (domail.php maybe, but I'll have to go back and look), those hacks seemed to get around this type of thing ever happening and they must have worked...

Now - back to the other question - DW - What about Bounce Processing?  It doesn't seem that resume.php makes sure that runs - my bounced mailbox is now at 60MB!    Is it ok to run Dailymail with ALL options turned off but bounce processing - will that do what I want?

[DW]

Thanks for your help on this one, Mike!

Brett, I must recommend that you have your web host manually delete all of the messages in the bounce mailbox.  I fear that the mailbox has become so full that checking messages in it could be disasterous.  Due to hard drive slowdowns with several thousand files in a folder it would probably take several days or weeks to process these messages.  I once did a test with 11,000 bounced emails and it took upwards of 6 hours.  This is why bounce.cgi is  strongly recommended for large lists.

resume.php does not process bounced messages - that's dailymail's job.  I'm actually a little surprised that dailymail is able to 'get past' bounce processing and queue messages even with such a large number of bounces in the mailbox.

Regards

[BGSWebDesign]

Hi DW,

This is why bounce.cgi is recommended for large lists.

It's not possible - think about it, with a large list - when doing a large mailing 30,000-40,000 think how many bounce requests come in simultaneously!  

When they do come in like that - they slow down the entire system and make it unable to handle simple requests - such as link tracking requests...  Now which would be worse - taking a little longer to process bounces - or losing sales because no one can click through on your links?

You can see the problem here - also - my question is still unanswered - WHAT happens when resume.php picks up where Dailymail started?  Is bounce processing EVER activated in this situation as it seems that it is not - so that means there's a problem with LMP in this situation - it may work with bounce.cgi - but it's impossible doing it the way I do - if resume.php will NOT run the bounce processing!

Regarding bounce.cgi - if you can show me with good testing/results that it's possible to run large lists such as I mention and still be able to process link tracking requests - I may consider switching - but until then - I'll need to find a way to do it on my own - because I absolutely can NOT lose sales when bounce.cgi is running every .5 to 1 second and sometimes more when large lists are mailed.

Let me know what you think and point me to some tests/results so I can determine if it's possible to switch to bounce.cgi for my environment.

[DW]

Bounce processing is never activated by resume.php - I hadn't thought about it being necessary.  Under ideal circumstances, especially if only used for smaller lists as recommended, the script will not fail and all bounces will be processed by dailymail.  With larger lists bounce.cgi should be used.

so that means there's a problem with LMP in this situation - it may work with bounce.cgi - but it's impossible doing it the way I do - if resume.php will NOT run the bounce processing!

The way you do is with too many bounces to a standard mailbox...  it is known that the pogram will not work as expected in this situation.  Try checking the mailbox manually and see what happens...

I absolutely can NOT lose sales when bounce.cgi is running every .5 to 1 second and sometimes more when large lists are mailed.

I totally understand.  What you can try is having your web host enable a higher 'maximum # of connections' to the web server.  Also, some MySQL optimizations can likely be made.  If this doesn't help you may want to consider a dedicated server - I know one user running a high-powered dual-processor server with 1 million addresses using bounce.cgi.

Until the process is improved you might want to (or have your host) set up a simple redirect to "null" for your bounce address so bounces can be temporarily discarded.

DW

[mike2]

Bounce processing is never activated by resume.php - I hadn't thought about it being necessary.

IMHO it shouldn't have to be, like you said in this case that's dailymails job and dailymail should pick up the next day and handle it.

However, like you said Dean, the mailbox has gotten WAY too full...  most email clients would choke on trying to download that much mail.  Actually the IPOP (or whatever POP3 program your server uses) process may not even be able to handle it in a shared environment.

Brett, I honestly believe in this case it is almost hopeless to get at that POP mailbox and the longer you let it go the worse it will get.  I'd cut my loses and overwrite it now to start fresh.

I totally understand. What you can try is having your web host enable a higher 'maximum # of connections' to the web server. Also, some MySQL optimizations can likely be made. If this doesn't help you may want to consider a dedicated server - I know one user running a high-powered dual-processor server with 1 million addresses using bounce.cgi.

This may or may not help, I believe (and this is from my own experience running a fairly big list of 150K) that the biggest slow down while this is going on is due to the amount of .CGI processes running taking up your resources.

Regarding bounce.cgi - if you can show me with good testing/results that it's possible to run large lists such as I mention and still be able to process link tracking requests - I may consider switching - but until then - I'll need to find a way to do it on my own - because I absolutely can NOT lose sales when bounce.cgi is running every .5 to 1 second and sometimes more when large lists are mailed.

I run a list of 150K and use bounce.cgi with no problems and I mail to the list daily.  Now granted I use two fairly low powered linux dedicated machines (on my own T-1 line) to accomplish this does make it a bit different.  One way that you could throttle this down is tweaking your SMTP program (which you never told me what it was) to only allow so many incoming connections at a time.  This is simple with sendmail...

Brett,  with that size of list(s), I HIGHLY suggest running your own dedicated server.  It's well worth the extra $100/month or so, but I also understand not wanting to have to manage one.

[DW]

Another user has reported what sounds a lot like the non-existent domain error in this post.  I'm offering tech support in an effort to be able to do some much-needed testing.

Brett, if you can wait a little longer I might be able to solve the problem and avoid the need for the custom hack I offered.

Regards

[BGSWebDesign]

Hi,

Brett, I honestly believe in this case it is almost hopeless to get at that POP mailbox and the longer you let it go the worse it will get. I'd cut my loses and overwrite it now to start fresh.

No, no problems here - I ran it manually - via Dailymail (turning off all other options) and it did just fine processing 7,800 bounces in about 20 minutes!  See the last message in this post:
http://listmailpro.com/forum/index.php?topic=1196.0

One thing I did notice - before I ran Dailymail bounce processing - I ran Mailwasher on the mailbox to see what was in there - I did notice that there were a few 'out of office' returns - I didn't think they were EVER supposed to be in there - and also a few virus potential mails - but either way - running Dailymail bounce processing did just fine... in a fair amount of time too.

Brett, if you can wait a little longer I might be able to solve the problem and avoid the need for the custom hack I offered.

That's fine, my own hack to fix admin.php so Dailymail can run Bounce processing by itself is just fine - I can manually start up Bounce Processing any time I want to handle this problem...

Acutally I believe the problem is related to something else - hacks I put into your domail.php module to STOP trying to connect every time when SMTP was returning errors - if you want I can share those with you - it seemed to make things work when I did that in v. 1.84 and apparently is what was causing the problem in this current version: I forgot to put them back in - when I upgraded!

[dean]

This is good information.  I do not believe this is as simple as "email addresses with two periods", however.  I did some tests and it seems that all domains can have quick or delayed DNS replies.  This is probably due to nameserver response time.

DW - Looks to me like DNS also.  The one I stalled on today was @ott.net.  When I send to the address I get an error back from the SMTP server.

      'test@ott.net' on 4/20/2006 2:42 PM
            451 DNS temporary failure (#4.3.0)

This website "ott.net" is currently not resolving via HTTP for me either.

:Dean

[dean]

set up test list with two users - one good, one bad

Here's the log:

> MAIL FROM: <email_handler@wesleyan.gotwww.com>
> RCPT TO: <test@ott.net>
250 ok
451 DNS temporary failure (#4.3.0)
LM: Undeliverable. RCPT response: 451 DNS temporary failure (#4.3.0)
. Skipping.
> RSET
451 DNS temporary failure (#4.3.0)

domail.php ends up here:
admin-26-Lost connection to MySQL server during query

[DW]

It seems that the server doesn't respond to the RSET command with success.  This must be the problem.  I will do some tests with the access info you provided and get back to you soon - thanks Dean!

Regards

[BGSWebDesign]

Hi,

This must be the problem. I will do some tests with the access info you provided and get back to you soon

Please tell us you corrected this, and when it will be released?

I still have the problem, this morning my mailing stalled on someone entering a dummy address:

Code: [Select]

test5@dsds.com obviously a bad address - but LMP chokes on it, I don't remember v. 1.84 doing that - I must have modified the code, but have looked for over an hour this AM for it and can't find what I changed - there is one thing with the bounce processing that was different but I don't think that is it...

UPDATE
======
I had a further look - I vaguely remember some problem with the '250' error?   Have a look at that, also there's that problem with the 'or die' code that was added... apparently I left it commented OUT and did not use it in my vers. 1.84 - here it is - you told me to ADD THIS and it fixed the problem - which I reported to you it did, but you failed to FIX it in the new version - why do I see this happening with you?   When someone reports a fix - you should immediately fix it in the PRODUCTION version, this is costing me serious time, here's the problem, line 2540 in admin.php, it needs to have the addslashes, like this:

Code: [Select]

$cmd = "select id,list,bounces from $utable where email like '".addslashes($email)."'";
$urows = mysql_query($cmd) or die('admin-45-'.mysql_error());

FURTHER UPDATE
===========
Surprise, I found this inside of one of my email texts - it must have been a problem at one time?

Code: [Select]

pipelining support enabled.
NOOPmsg=250 2.0.0 OK

PIPELINE-FROMmsg=250 2.1.0 ... Sender ok .
RCPT TO: <>
RCPTmsg=553 5.0.0 <>... User address required
DATAmsg=503 5.0.0 Need RCPT (recipient) .

Maybe that can help you DW?

PLEASE fix that in the production version!  And then look for '250' errors and fsockopen problems - I know I patched up those on my own too - but can't seem to locate them at this time...

DW - do you have a fix yet?

[mr.trevor]

Excuse my ignorance but... how do you get someone entering a dummy address and it being added to a list? If they do not exist they won't get confirmed. Are these not dropped after not being confirmed?

[dean]

how do you get someone entering a dummy address and it being added to a list? If they do not exist they won't get confirmed. Are these not dropped after not being confirmed?

Only if you have (and you should) confirmation turned on.

[BGSWebDesign]

Hi,

how do you get someone entering a dummy address and it being added to a list?

You got it, I don't use confirmation.

DW - please see 2 messages above where I lay out what I think is causing the problem...  

Please keep us informed on when you will have a fix for this.