Problems with Scheduled Emails stopping and Server Load

[don1r]

Hi Dean,

I am having two problems with ListMail (I am using version 1.77b).

1. The cron job "scheduled emails" function randomly stops, which means I have to manually send to the rest of the list when I login to check on the mailing. My web host said I should try changing the setting in the dailymail.php from - ini_set("max_execution_time","0"); - to - set_time_limit(0); - but I want to know what you think before it try that. Any suggestions?

2. When manually sending emails to my list using the recommended SMTP settings ListMail puts a huge load on the server. I can login to my control panel and watch the "Server Load" go from an average of .67 (less than one) to over 7 (seven) just from sending to 250 people. If I send to my entire list of over 3,000 people it crashes the server. My web host says that ListMail is opening and closing a SMTP connection for every email that it sends which overloads the server. Is this true? I don't understand how only sending a few emails can be such a drain on the server but I have witnessed this several times by watching the "Server Load" when I am sending emails. Any suggestions?

Also, the login.php for version 1.77b does not capture or send the IP address of potential intruders so I added the following code from the old version (1.77):

   $ipaddr = $_SERVER['REMOTE_ADDR']; if(!$ipaddr) $ipaddr = $REMOTE_ADDR; if(!$ipaddr) $ipaddr = "Unknown";
   $refurl = $_SERVER['HTTP_REFERER']; if(!$refurl) $refurl = $HTTP_REFERER; if(!$refurl) $refurl = "Unknown";
   $iphost = @gethostbyaddr($ipaddr);

Your assistance with my problems listed above would be appreciated.

Thanks,

Don

[gluntz]

Hi Don (and Dean),

I am seeing excruciatingly slow execution on my Send Email run.  No matter whether I use PHP or SMTP, it halts after 200 or 250, then just sits.  In order to get ANY forward movement, I have to continually hit the Click Here button.  I do expect a delay after 200 or 250 (depending on PHP or SMTP settings I have). However, it does not say "Delaying" - it just sits. I left it for 2 hours and still saw no movement.

Obviosuly, sending to 15,000 recipients makes this unusable.  Any ideas?

[DW]

1. The cron job "scheduled emails" function randomly stops, which means I have to manually send to the rest of the list when I login to check on the mailing. My web host said I should try changing the setting in the dailymail.php from - ini_set("max_execution_time","0"); - to - set_time_limit(0); - but I want to know what you think before it try that. Any suggestions?

It's possible that an address on your list is causing a problem with the SMTP process.  When you resume your mailings, do they go out fine?  If so, then this likely isn't the case.  The scheduled task utilizes the "-T 0" paramater (as per http://www.listmailpro.com/help/10.html), which should prevent any "wget" timeouts.  I am not sure if using "set_time_limit(0)" will be better than the method i use - few, if any, people have problems with the method I use.  I'm not too sure how to fix this one.

2. When manually sending emails to my list using the recommended SMTP settings ListMail puts a huge load on the server.   If I send to my entire list of over 3,000 people it crashes the server. My web host says that ListMail is opening and closing a SMTP connection for every email that it sends which overloads the server. Is this true?

No, ListMail does not open a new SMTP connection for every single email.  That is, unless you have "Reconnect Every X Emails", on the config page, set to 1.  Default is to reconnect every 2500 emails.  ListMail sends all email from a single SMTP connection.

It makes more sense that "Internal PHP mail()" might crash the server, since with this method of sending a new process is spawned on the server for each individual email.  With SMTP, however, all email is "queued" to the SMTP queue and then delivered as the mailer chooses.  Many web hosts can deliver queues up to several hundred thousand with no problems.

It is highly recommended that you use SMTP only for sending and that you make sure your "Reconnect every X emails" setting on the config page is set to at least 1000.

I'll check into the IP address problem with login.php, thanks for the heads up.

[don1r]

Hi Dean,

Here is a follow-up on my previous post.

It's possible that an address on your list is causing a problem with the SMTP process. When you resume your mailings, do they go out fine?

When I resume the mailings they do go out fine except for the problem of overloading the server that I mentioned.

Also, why would a bad email address stop the "scheduled emails" process? Can this be fixed? It seems that this should be a top priority because people often change and/or discontinue their email addresses or enter the wrong one. Also, I am using the bounce.cgi script.

As far as the manual sending overloading the server goes:

I am using the SMTP Server to send (not "Internal PHP mail()") and I am using the "localhost" setting with "no" AUTH and "no" "POP checking" authorization. I have the "Reconnect every X emails" set at the default setting (2,500).

Question: If the AUTH function is not checked does the "Reconnect every X emails" function still work or does it only work when the AUTH function is checked?

I also noticed that my manual mailings seem to be going out immediately (they do not appear to go into a queue). Do you think that this could be why my server gets overloaded? From just sending 250 emails the server load will go from less than 1 (one) to over 7 (seven), when the server load hits 10 (ten) it crashes the server. How can so few emails be putting that much of a load on a huge web server? The only way that I can safely send is to stop the process every 300-400 emails and wait for the server load to go back down to below 2 (two), which only takes about a minute but the whole process is becoming a big hassle.

Do you think I need to add some sort of throttle script/code to space out the flow of the emails or should I have my web host check to see if the mailing queue is set up properly? Or, do both?

Thanks,

Don

[DW]

Also, why would a bad email address stop the "scheduled emails" process? Can this be fixed? It seems that this should be a top priority because people often change and/or discontinue their email addresses or enter the wrong one.

This was a bug in a previous version.  If a local address (one on the same server) did not exist ListMail would become confused with the SMTP responses because instead of sending a bounce the server would respond directly in SMTP.

Also, I am using the bounce.cgi script.

If a lot of your addresses are bouncing it can cause some extra load on the server.  Try disabling your ListMail bounce handling, but make sure your Administrator Email is one on your ListMail domain so that it can be properly set by ListMail as the "Return-path" (bounce to address).

If the AUTH function is not checked does the "Reconnect every X emails" function still work or does it only work when the AUTH function is checked?

Sorry about the confusion.  "Reconnect every X emails" works with or without AUTH enabled.

I also noticed that my manual mailings seem to be going out immediately (they do not appear to go into a queue).

This would be up to your host's SMTP settings.  If your host cannot send email as fast as you queue it to the server, it will 'build up' in the server queue.  This few emails should never be able to crash a server.  In fact, no amount of emails should crash anything - it should simply slow it down.  Your host may have some sort of failsafe that shuts you down if you use too much load- though, this is rare.  Is the entire server going down, or just your site?

Your host could optimize their mailing software for larger amounts of email.  They may be using one of the mailers noted in the Optimization section, at http://listmailpro.com/forum/index.php?board=8 . Also, they could increase the MaxClient setting in Apache to 255 for better bounce.cgi handling, as it goes through the web server.

[gluntz]

Dean,

I sent out a reminder to my 16,000 names last night.  Same problem .. I spent 4 hours "nursing" the process, hitting the "Click Here" ever 150 - 200 emails.  This is obviously not workable for us.

I am not sure if this gives a hint of the issue or not, but I notice that:

a.  After 150 - 250, the process "completes"; by that, I mean the progress bar at the bottom of the browser moves from 50% or so to fully complete and disappers;

b.  This is not the same as when the system proactively "delays" after the preset number of emails, give the delay maessage and then starts up again. In those instances, the progress bar at the bottom of the browser never finishes, it inches forward as the number of emails sent edges towards the 16,000.

I am not sure if this is indicative of an issue that you might recognize; but, we do need a solution to this.  In the absence of a solution, we will have to find an alternative way to handle this.  With a list growing at over 2,000 identities a month, this has now become a priority.

[DW]

It sounds like your server may have a strict time limit on running scripts.  We could run a test with a simple test script, like the following, which will output data every 1 second for up to 2 minutes:

Code: [Select]

<?php// try to set the time limit to unlimitedini_set("max_execution_time","0");// loop for 2 minutes$stop_secs = 120;$start_time = date("U");$done = '';$old_elapsed = '';echo "time elapsed=";// loopwhile(!$done){ $elapsed = date("U")-$start_time; if($elapsed<>$old_elapsed){  echo " $elapsed..";  $old_elapsed = $elapsed; } flush(); if($elapsed==$stop_secs) $done = 1; }echo "time's up ($stop_secs secs), seems ok..<br>";?>

Copy and patse the code above into a PHP file, ie: "test.php".  Then, upload it into your public_html folder and browse to it to activate..  If the script stops before the 2 minutes is up you likely have a server that will not allow scripts to run for as long as necessary for ListMail.