Unexpected SMTP delay or failure with some domains

[DW]

These SMTP errors are one of my highest priorities - I've been working on it but haven't been able to fix it yet.  I should have something soon.  I am not sure if these problems are related to the addslashes() fix for the bounce function, but I will look into this too to make sure it's in the next update.

Using ListMail's methods you should not be able to enter email addresses in a bad format (ie. blank) into the program.  This is usually the result of using a custom script or plugin to insert users.

From numerous Google searches I am led to believe the "451 DNS temporary failure" error is related to optional qmail-specific patches.

I am working on allowing SMTP to recover from this error and skip the user during mailout.

Regards

[DW]

Brett, is there any way you can show me your LM SMTP log when the mailing stalls?  On Dean's server, which have access to, I'm getting a new type of error "lost connection to MySQL".  I am not sure if this is the problem, part of the problem, or a side-effect of the problem. You may need to uncomment the line "$smtp_debug = 1;" in admin.php to enable global SMTP log  so the log is written during dailymail, resumes, etc.  In the meantime I will continue troubleshooting this with the access I have, of course.

Update re: bounce() addslashes:

This was added to the top of the bounce() function in admin.php to eliminate the need for the change I suggested to you previously:

if(!valid_email($email)) return false;

Regards

[dean]

On Dean's server, which have access to, I'm getting a new type of error "lost connection to MySQL

When this first started happening I asked my host about this.  They said it was just the standard time out error. "This happens when a query is using too much MySQL processing time.  If a query takes longer than 10 seconds mysql will drop the connection." They said this has been in place for over 6 months and had not previousley effected LM.

Every time I get this error the SMTP log also logs the DNS error.

:Dean

[DW]

I have found some interesting information, such as:

http://dev.mysql.com/doc/refman/5.0/en/gone-away.html

"Lost connection to MySQL server during query"

I've found that this error also will occur if for whatever reason, the client takes longer than <connect_timeout> seconds to connect to the server. The default value is 5 seconds. You can increase this value in your my.cnf file or elsewhere, though it would be ideal to figure out why it's taking so long to connect.

This may be what is happening, caused by the delayed (~10s) response from the DNS lookup.

Brett, it would be interesting to know if you also receive the error message Dean does:

admin-26-Lost connection to MySQL server during query

This message should be displayed on the sending page when the mailing 'stalls'.

I am performing more tests to verify this.

Regards

[DW]

Indeed, when I manually configure admin.php to simply skip to the next email on a certain failing address the MySQL connection error message does not appear and sending continues as normal.

I am not sure what I can do about this.  Possible solutions I can think of are all based on server configuration.

1. A line could be added to your /etc/my.cnf (MySQL config file) to change the connect_timeout variable from 5s to 15s (which I think should be long enough).

Code: [Select]

[mysql]
set-variable = connect_timeout=15

2. The SMTP server could be configured to not perform the DNS lookup on-the-fly during the SMTP "RCPT TO" command.

I am assuming this is qmail-specific problem because searching Google for the error "451 DNS temporary failure" results in a number of results containing optional qmail "patches".

Some relevant information may be able to be found here:
http://web.infoave.net/~dsill/lwq.html#smtp-slow

Some possibly irrelevant qmail options can be found here:
http://web.infoave.net/~dsill/lwq.html#config-files

A list of "RCPT TO" patches, one of which could be causing this lookup on-the-fly/problem, seems to be listed here (though, the majority these patches seem to be used to validate local addresses:
http://http.netdevice.com:9080/qmail/rcptck/

Work-arounds might include:
1. LMP recipient email DNS checking before a mailout.

This is a tough one..  I hope to be able to find a specific patch that is causing the problem and be able to offer a solution soon.

Regards

[DW]

My host says they have the time out set to 10 but it has been that way for 6 months. The 1.84 version is where this started.

Also, the domains that are causing the problem ARE failing dns look-up (not just at the host) - so they are the problem.

Can't you just "if(timeout) skip"

The process has not changed in recent versions. Skipping may be possible but it won't be easy.  The delay occurs when waiting for the server's response to the SMTP "RCPT TO" command.  I will see if I can place some sort of timer on the read function so you can set how long to wait for response to these requests.

Regards

[BGSWebDesign]

Hi DW,

I will see if I can place some sort of timer on the read function so you can set how long to wait for response to these requests.

This seems the most logical way to do it, please let us know when this is available...

Regarding v. 1.84 - I believe it may have started there - I know that I added/updated the code throughout your loop that sends email to make sure it skipped bad email addresses sometime before this version - I never had this problem in v. 1.84 - I believe it's because of the code I put into your loop, but I can't seem to locate the code - I spent hours looking on Friday, if I find it, I'd be happy to share it with you - I'm pretty sure I was looking for bad addresses (via the error codes - 550 or some other number) and then skipping the re-connect or other code at that point and continuing on - can you locate that code?   If so tell me where it is, it will help me track down the changes I made...

UPDATE
=====
I think I found something, I went back to v. 1.74b code and found all of these sections within the send link where it tried to bail out, for example here's one:

Code: [Select]


     if($lastmsg == '503'){
         fputs($ssock, "RSET\r\n");
         $rsrvmsg = fgets($ssock,1024);
$errmsg .= $srvmsg;
         if($bugs){ echo "RSETmsg=$srvmsg<br>"; flush(); }
         flush();
         $skiptonext = 1;
         $reconnect = 1;
         if($bugs){ echo "Bad RCPT response ($email), skipping.<br><br>"; flush(); }
     } else {


This is inside of this section of code (in v.1.86):

Code: [Select]


$lastmsg = substr($srvmsg, 0, 3);
           if ($lastmsg <> "250"){


Approximately - line 1500, hth, let us know what you find, I also had some code in there to look for if $email= '', do you need that also?

[BGSWebDesign]

Hi DW,

Ok, I had to re-work the code inside the send loop throughout admin.php, I added code for '503' error code and made sure that every time I found an error - or unable to send I set these two:

Code: [Select]


$skiptonext = 1;        
$reconnect = 1;


That's about it, not bad really, but you'll probably want to play around and test it - btw do you have a decent test bed DW?   It doesn't sound that way if you're not able to duplicate these types of hiccups?

If you need the code email me...  I can post it up here, sometime tomorrow, I'll have to go line by line as I re-worked the send loop in about 3-4 different places.

(*see previous message in this thread also)

UPDATE
=====
Nope, that's not it, it's just stalled on me again - ok DW, since I've been working in that send loop - can you please just tell me WHERE to put some code to delete the top record in lm_sendq - so that I can just delete that record and continue (for example - if skipnext, perhaps look at the record count - reccount, and if it's the same just delete the record)?  

Please tell me, or provide the code, that's all we need - this is a pain, it's stalling every day!

2ND UPDATE
========
Ok, it's been 10 days since I first posted the first thread - when are we going to get some code to fix this issue?   I'm going to code it myself tomorrow if I don't find something up here - I'll save the reccount() in lm_sendq, if it's the same several minutes later - I'll delete the top record, that should about do it - and I'll put it in the looping code - I don't see what the problem is with providing code to do that - or building it into resume.php - more likely it belongs in the send loop within admin.php...

Skipping may be possible but it won't be easy.

 Why?  I don't see what's so hard about it, really, and I didn't write this code, you did...  Is there something I'm not thinking of, as from what I can tell it only involves a simple reccount() check in the send loop (as I mention above), if it stays the same for several minutes, obviously there's a problem with sending... so delete the top record in the table lm_sendq - can you fill me in DW on anything else?

[DW]

You're right, of course, Brett - there was no need for the pessimism.

Ok - I got it!  Change the function "getsmtpmsg" in admin.php (search for "ion get") to the following:

Code: [Select]

function getsmtpmsg($xsock){
 global $smtp_timeout;
 if(!is_numeric($smtp_timeout)) $smtp_timeout = 9;
 $data = "";
 stream_set_timeout($xsock,$smtp_timeout);
 while($str = fgets($xsock,1024)){
  $data .= $str;
  if(substr($str,3,1) == " ") break;
 }
 return $data;
}
To override the default setting of 9 seconds, set the $smtp_timeout var in config.php.

There was some additional cleanup to do in the sending routine (mostly log file writing, etc), but the above change will fix the immediate problem with the same working result.  The code in all it's glory will appear in v1.87.

The timeout, unfortunately, renders the socket connection useless and a reconnect must be issued.

I plan to include a tool to weed out these non-existent addresses before they are emailed ASAP.

I also expect to issue a critical release very soon in light of the recent bounce mailbox bug discovery.

Ok, it's been 10 days since I first posted the first thread - when are we going to get some code to fix this issue? btw do you have a decent test bed DW? It doesn't sound that way if you're not able to duplicate these types of hiccups?

Since individual servers can vary so greatly testing can require that a user give me access to their installation.  If you could have provided a test installation for me to work on I might have been able to start sooner.  Thankfully, another user having the same problem (dean) came to the rescue by submitting his info.  This fix was made possible by having direct access to his installation and files.

I hope this fixes your troubles, Brett!  If it doesn't, try setting the timeout lower, with a line such as follows in config.php:

Code: [Select]

$smtp_timeout = 4;

Regards

[mr.trevor]

I plan to include a tool to weed out these non-existent addresses before they are emailed ASAP.

Hi DW, I have been following this thread (as far as I can understand it...) with interest.

As well as a 'tool' for cleaning up existing lists (if I understood that correctly), will something be included in the subscribe code to check and prevent these from being added to a list in the first place, maybe.? (possibly even checking any added manually...)

(My understanding of a 'tool' is something that you use when needed, as apposed to something inherent in the system that you don't have to remember to use.)
Regards.

[DW]

I was thinking the routine could be run manually and/or before dailymail each day.  I like the idea of adding it to the subscribe process - I'll definitely set that up along with a new error message.  This should probably also be added to the import process.  It might be better to run the verification after import rather than on-the-fly... either way it should work.

Check out this very useful PHP code snippet:
http://php.net/manual/en/function.getmxrr.php#64117

It seems from the resulting SMTP responses we might even learn the exact status of mailboxes, such as "Mailbox Full".  That is, if there's a numeric SMTP code for each status, as the text messages are mailer-configurable.

The question is, do we want to do this on EVERY mailing for EVERY message and ultimately take ALL unnecessary load off of the SMTP server and bounce routines?  Sounds like a good idea to me.

While I'm at it, why even send to an SMTP server at all... I could include a new mailing option for ListMail to be it's "own" SMTP server.

The most concerning problems, I think, are 1) the number of sockets we are able to open up at a time and 2) achieving some sort of efficiency/speed. I think common SMTP software has come a long way in this area...  Then again, it wouldn't necessarily be slow overall, because ListMail needs to take the time to process the queue, message codes, etc. for each email anyways.

I'm going to move this to a new thread soon for more discussion.

UPDATE: I'm done editing this post now.

Regards

[BGSWebDesign]

Hi DW,

Ok, back to this - thanks a million - it did solve the problem, oops I think I spoke to soon...

UPDATE
======
The change to getsmtpmsg has slowed down my sending considerably - is that supposed to happen DW?  I tried it with $smtp_timeout=9, it's slow, so I moved it to $smtp_timeout=4 as you suggested and it's even slower, so now I'm going the other way $smtp_timeout=12 - WHAT am I affecting here, it seems like it's stalling about every 30-40 emails now???

Another thing I still wonder about though, I seem my mail going through, but recently using the new function you wrote (getsmtpmsg($xsock)) - it seems that I'm getting fewer clicks through, and fewer purchases...

What I'm wondering about is if this new function would be somehow NOT getting some of the mail through because of this:

Code: [Select]

while($str = fgets($xsock,1024)){?

WHAT if the 9 second delay is not enough to get 1024 bytes through?  I'm guessing that it would SKIP this email address and go on to the next is that right?

-----

Regarding your other interesting find:

It might be better to run the verification after import rather than on-the-fly... either way it should work.

 

YES!  Please provide a way to run this separately AFTER import, and also give anyone a way to turn this Off/On when they want to, I'm not 100% sold on this idea - as I would guess other large list members up here are also not...

Here's why - this needs to be fully tested, as does the little getsmtpmsg function described above, in a full production environment with a LARGE list to be absolutely sure - 100% POSITIVE - that the same number of emails will get through with these new function calls in place.

The question is, do we want to do this on EVERY mailing for EVERY message and ultimately take ALL unnecessary load off of the SMTP server and bounce routines? Sounds like a good idea to me.

I'm skeptical, as I've seen these type of things before and they never seem to work completely, or at least it may LOOK like they are working, but try them out on a large list and you lose 1/2 of your messages going through!  Yes, it can happen!  So, please see to it that you full test these functions on a very large list and make sure you are getting the correct numbers, for all of us, then please post the results so we can all rest easily before this type of thing is put into the code in the current version...

While I'm at it, why even send to an SMTP server at all... I could include a new mailing option for ListMail to be it's "own" SMTP server.

Now that sounds real promising - but I'm guessing if you're going to do something like that - it ain't gonna work in Php, it's just not fast enough, you're probably going to need Perl, or a compiled 'C' program as a base module to pull that off???   Best of luck, interested to hear what you find out and further information on this moving forward...

[mike2]

If you want my $0.02 as a somewhat large list owner...

I'd much rather Listmail NOT do the SMTP work at all... thats what (~INSERT YOUR SMTP PROGRAM HERE~)'s job is.

IMHO all the problems you are having with this is probably more easily fixed by tweaking your (SMTP PROGRAM) instead of Listmail.  

Although I understand, Dean, that you want Listmail to "Work" with everything.  

As I stated before, it should be this easy:

Listmail generates the email, connects to SMTP, sends it, Email program says ok, next, etc, etc.    THEN the SMTP program tries to deliver the message and "bounce" processing removes any bad messages.

Of course on signup/import, etc.  I think the rules in place already for simply not allowing badly formed email addresses is probably enough.  I think it probably would be a BAD idea for some kind of super complicated routine to make sure an email address is actually working when imported.

[BGSWebDesign]

Hi DW,

Please reply to this - you never responded to my previous message (above Mike's message)....

It seems that adding this new function (or adding the delay) to the smtp calls is helping get the mail delivered but it SLOWED down my emails tremendously, it's taking 2-3 times longer to get my mail sent out!

I now have $smtp_timeout = 12;  set like this, I tried original suggestion of 9 and had problems, so went to 4 - that did NOT work and still hung up - so I went back the other way to 12 - but now my mailings are very, very slow....  

What is going on here DW?  Is there any way to get this under control, I still think the best way to handle this is to do what I suggested, you put a SQL call in resume.php - it looks at the reccount() in lm_sendq, if it does NOT change in 30-40 seconds, delete the top record - that way the mail can still go through quickly - instead of throwing this LONG DELAY into every single SMTP call - think about it.... let me know what you plan to do, but I know I can NOT tolerate this slowness of mailing, I'll have to go back and write my own function call in resume.php if you can't supply the code for it... or at least give me an idea on how to do it - how much time to wait, etc...

[DW]

It seems that adding this new function (or adding the delay) to the smtp calls is helping get the mail delivered but it SLOWED down my emails tremendously, it's taking 2-3 times longer to get my mail sent out!

This is somewhat expected as it will take up to the $smtp_timeout # of seconds for each invalid address to fail and ListMail to detect it, skip the user, and reconnect.  If you get several of these in a block of 50 you might notice your counter incrementing very slowly.  What we need to do is clean your list with a script such as mentioned here to remove these failing addresses from mailing entirely.

if it does NOT change in 30-40 seconds, delete the top record, instead of throwing this LONG DELAY into every single SMTP call

We should be able to continue within 5-10 seconds with the changes made.  There is no mandatory delay for every call, a 'timeout' was simply added for when the server doesn't respond sooner.

Regards