Scheduled mail, is it working - Throttle Code

[BGSWebDesign]

Hi,

Love this new board...  ok, quick question, I need an answer fairly soon as I'm about to do a re-send.  

I have a huge list 120,000 that I've split up into pieces of 10,000 at a time usually.  I did a 'scheduled mailing' for the first time last night.  I see some clicks coming in from click tracking, but it doesn't seem to be enough, or match the number I should see for a 10,000 piece mailing?

Can you please tell me the database field that is used when a mailing is done to track if that piece has been sent?  

Or, is there some other way to know if a user in the user database has received that scheduled mailing?  Right now that is the ONLY mailing going out to that list.

Thanks in advanced,
-Brett

[DW]

Neil,

I wonder if this is the same problem you are having where you reported a low number of removes.  It is possible that your server has not finished sending your email.  Some hosts are not optimized for sending very large amounts of email quickly and can get backlogged considerably, taking hours upon hours to finally finish delivering your messages after ListMail queues them.

When sending an email you can check out the number of rows in the lm_sendq table, which is the ListMail queue.  When the program starts sending to the server you should see a large number of entries in the queue.  As messages are sent to the server the number of items in the queue decrease.

One way to see if the server is indeed accepting your messages is to uncomment a line in admin.php to enable SMTP debug messages:

 // debug, shows SMTP messages
 // $bugs = '1';

should become:

 // debug, shows SMTP messages
 $bugs = '1';

Note that this will output a LOT of SMTP messages when sending email to your list.  These messages can only be seen when doing a "Send Email" to your list, not with scheduled email.

ListMail should, however, detect when your server is not accepting messages correctly..  It seems likely that your host is having troubles sending the messages once queued.

[BGSWebDesign]

Hi,

I wonder if this is the same problem you are having where you reported a low number of removes.  It is possible that your server has not finished sending your email.  Some hosts are not optimized for sending very large amounts of email quickly and can get backlogged considerably, taking hours upon hours to finally finish delivering your messages after ListMail queues them.

Yes, this is the same problem.  When you say 'backlogged', does that mean that the email does not get queued at all?  I'm not certain that this is happening as I 'seeded' my user database with my own personal Email addresses on one of my lists (at the beginning and the end of the list) and I did receive the Emails, leading me to believe that the Emails are getting delivered, would you agree?

When sending an email you can check out the number of rows in the lm_sendq table, which is the ListMail queue.  When the program starts sending to the server you should see a large number of entries in the queue.  As messages are sent to the server the number of items in the queue decrease.

When the number of items in the lm_sendq table is greater than zero, would I receive a notice if I logged back in and clicked on the Send Scheduled Email?   I expected that I would see something on that screen stating that Emails were still in the queue, is that true?

Note that this will output a LOT of SMTP messages when sending email to your list.

Is it 'ok' to turn bugs=1 with 10,000 users in the database?  Where do the SMTP messages appear?  You say it will output a LOT of SMTP messages, where?  To the screen, or to my Email address?  If I have it turned on, what should I look for, will I be able to look at everything and fine tune it with 10,000 emails being sent out?   Would it be easier to break up my User Database to 1,000 size and then turn bugs=1 and see what happens, so I can make adjustments?

[DW]

When you say 'backlogged', does that mean that the email does not get queued at all?

No, what I mean is that the server will accept the email from ListMail but then email would be stuck in the mailing software (sendmail, qmail, postfix, Exim, etc.) queue waiting to go out.

When the number of items in the lm_sendq table is greater than zero, would I receive a notice if I logged back in and clicked on the Send Scheduled Email?

When you login if there are items in the queue table you will automatically see an addition to the header of ListMail stating that there are unsent items in the queue.   You can then "Resume" or "Cancel" a certain mailing.  Since you can login while a mailing is in progress, you should first determine if the mailing has stopped by moving around the program and seeing if the "Remaining messages" number is decreasing.  The "Scheduled Email' feature does not play a part in this.

Is it 'ok' to turn bugs=1 with 10,000 users in the database? Where do the SMTP messages appear?

It is okay to turn on the debug feature with 10,000 users but will use a lot of bandwidth.  Hopefully, you are on a high speed connection.  If you get tired of seeing the messages you can simply close the window and re-login to ListMail to resume the mailing after you have disabled the debug messages.

The messages appear on the send email page underneath the counter.

[BGSWebDesign]

Hi Dean,

Love this support forum, it's great, thanks!

When you say 'backlogged', does that mean that the email does not get queued at all?

No, what I mean is that the server will accept the email from ListMail but then email would be stuck in the mailing software (sendmail, qmail, postfix, Exim, etc.) queue waiting to go out.

Sure, but if it's stuck in the Email it would eventually get delivered, right?
In other words, please answer this previous question:

I 'seeded' my user database with my own personal Email addresses on one of my lists (at the beginning and the end of the list) and I did receive the Emails, leading me to believe that the Emails are getting delivered, would you agree?

Then, please answer the next questions...

When you login if there are items in the queue table you will automatically see an addition to the header of ListMail stating that there are unsent items in the queue.

 

The next time I logged in was probably 7 to 10 hours AFTER the scheduled Email started to go out. I NEVER saw that header, leading me again to believe that ALL the Email was sent out, right?  

It is okay to turn on the debug feature with 10,000 users but will use a lot of bandwidth.  Hopefully, you are on a high speed connection.

I am only on Dialup right now!  That won't work, thanks for letting me know, trying it with 1,000 will that still be a problem with output?  What about 100, or 500?

[DW]

I'm glad you like the forum.  Thank you for participating!

If you do not see anything in the queue then it should be safe to assume that your message went out.   I recommend making sure you are on v1.77b, though, since there is much more efficient SMTP error checking.  Previously, if a connection was lost or broken in a certain way ListMail might not detect it.  

Messages being sent from ListMail to the server do not guarantee delivery.  After email is queued to the server, it is up to the web host to finally deliver the messages.  Some hosts are configured to send it right away, as fast as you can queue it.   Others are set to delay for an hour.  Still others might get backlogged and you never know if the host has cleared the queue in an attempt to remedy things, which could cause you to lose messages.

[DW]

I would go ahead and run a test to up to 1000 users with debugging information enabled just to make sure messages are being accepted properly by the server.  If they are, your messages should be guaranteed delivered.

It's possible you have a low number of removes because people are afraid to click the "Remove Me" link on the remove Custom HTML page.  I know I usually am!  Perhaps an addition to the "Remove Link" message code, to allow for 1 click removal, is in order.  I would also make sure that the text you have above your remove link is friendly and professional.

[BGSWebDesign]

Hi,

It's possible you have a low number of removes because people are afraid to click the "Remove Me" link on the remove Custom HTML page

Ok, I just went back and looked.  One group of 10,000 users has NO removals, another has 2, and another 33, something very strange here.  I believe that the Emails are getting to the queue and then the queue is getting cleared by Admins, or automatically from the Host.

I host on a shared server, and I'll bet many others do to.  That is why I have to request again that you consider adding a Throttle option to the sending operation.  Make it a setting in the Configuration, so that you can turn OFF, or ON the Throttle, OFF by default, then allow the user to Set how many Emails are sent, and how long the Pause is between throttle counts...  For example: Send 100 Messages, Pause 25 Seconds, like that.

I'm going to run my own sending script (written in Perl, and that only accesses comma delimited text files - that has throttle code in it that I wrote) and see if the Removes are greater, I'm sure they will be, that will confirm that the Throttle is required.

Please tell me where in the code to find your global Sending Loop so I can try testing by inserting my own Throttle code and see how LMP does with Throttle code in it.

[DW]

In admin.php you will find the domail() function.

Look for this line:

Code: [Select]

   } else {
     // use php mail()
Above the else statement is the SMTP loop.  You can place your delays right before the text above.

the $y variable contains the current msg number.  To delay 25 seconds every 100 emails try this:

Code: [Select]

if(($y % '100')==0 && $y <> sizeof($sendq)) sleep(25);

[BGSWebDesign]

Hi,

Ok, regarding the use of a throttle with the code here from your previous message Dean:

if(($y % '100')==0 && $y <> sizeof($sendq)) sleep(25);

That did nothing, as I received the same number of clicks regardless of whether I used the throttle or not, so that's not the problem.

Back to the other issue, the number of removes does not make sense.

Here's some INTERESTING NEWS!  I'm not 100% sure on which batch of 10,000 I had the throttle turned on for, but here's what I see between 2 batches, one had the throttle on, the other was with the throttle code removed and off, the number of removes in one of those batches is 128, that's the highest number yet, check out the other number of removes in these batches:

Batch - Batch Size - Removes
 A   -    10,000   -     17
 B   -    10,000    -    24
 C   -    10,000  -      18
 D  -     13,000   -     31
 E   -      8,000   -     43
 F   -     10,000    -     2
 G   -        100      -    3  (with SMTP trace bugs=1)
 H    -     1,000   -     14  (with SMTP trace bugs=1)
 I    -    10,000   -       1
 J   -     10,000   -   128  (with throttle code, delay 25 every 100)

Do you see the pattern here!  When the code is slowed down, either by using SMTP tracing bugs=1, or by using the throttle code, the number of removes increases dramatically ,why is that?  Is it because when the sending is going fast there are too many REMOVE clicks coming in at once for the script to handle it?  It appears that could be the case, I've not had many people email me and say they have problems, but a few have emailed saying that the REMOVE code link does not work???  

Here are sample outputs from my SMTP bugs=1 code, as you can see, it looks like everything is normal here, see anything wrong?:

Code: [Select]


250-ENHANCEDSTATUSCODES
250-PIPELINING
250-8BITMIME
250-SIZE
250-DSN
250-ETRN
250-DELIVERBY
250 HELP

pipelining support enabled.
NOOPmsg=250 2.0.0 OK

Delaying.. Please wait..
If your mailing stops for any reason, close the window or . Do NOT Refresh!

Sending Mail  of 101..
PIPELINE-FROMmsg=250 2.1.0 ... Sender ok .
RCPT TO: <xxxxx@domainname.com>
RCPTmsg=250 2.1.5 ... Recipient ok
DATAmsg=354 Enter mail, end with "." on a line by itself .
SENDmsg=250 2.0.0 i45JbvSH023567 Message accepted for delivery .

PIPELINE-FROMmsg=250 2.1.0 ... Sender ok .
RCPT TO: <xxxxx@domainname.com>
RCPTmsg=250 2.1.5 ... Recipient ok
DATAmsg=354 Enter mail, end with "." on a line by itself .
SENDmsg=250 2.0.0 i45JbvSI023567 Message accepted for delivery .

PIPELINE-FROMmsg=250 2.1.0 ... Sender ok .
RCPT TO: <xxxxx@domainname.com>
RCPTmsg=250 2.1.5 ... Recipient ok
DATAmsg=354 Enter mail, end with "." on a line by itself .
SENDmsg=250 2.0.0 i45JbvSJ023567 Message accepted for delivery .

QUITmsg=221 2.0.xxxx closing connection .

Finished sending!


Well, again, as stated above, it appears when the code slows down, and the throttle is on, or the number of sends is slowed down with bugs=1 turned on the number of removes increases considerably, it seems a sure sign that on high speed, something is not working right when people try to remove themselves, or all of the emails are not getting sent out, though that really does not seem to be the case as I see almost the exact same number of clicks regardless of whether throttle is on, bugs=1, or normal sending!

Here's one more thing to think about.  I did update my admin.php when I added the throttle code, is there anything in there which would affect the performance of the removes?   If so, perhaps the udpated admin.php fixed the remove problems, I'll know soon enough when I start sending more batches, if I turn the throttle code off and still see a good number of removes I'll guess that the new admin.php solved the problem.

What are your thoughts Dean?

[mike2]

I have a suggestion that I did on mine - I have a big list around 150K...

I noticed that each time a person clicks to remove that the DB is optimized, so if 5 people click it at once, 4 of them will be waiting a long time and may never let it finish...

I commented out the optimize part in the rem.php file, I really didn't see a huge need for it, since dailymail will optimize the table daily and I haven't had any problems since.

So if you think that may be the problem, look in the rem.php and do something like this:

Code: [Select]

// MJR Rem'd out the line to not optimize on a remove
// mysql_query("optimize table $utable");


Michael Rogers, President - M&A Computer Services
Get Extreme Marketing:  http://extrememarketingprogram.com
Get Extreme Leads:  http://extremeleadprogram.com

[DW]

Ahh yes.  I should have thought of this as it was noted a while ago and should be fixed in the currently available v1.77b files.

[BGSWebDesign]

Hi,

That has to be it!  I just looked and the older files I was using had the Optimize command in there, the newer updated ones do not have it, and that must explain the greater number of removes!

Thanks, I'll keep you posted when I do a send on the next groups and see how they turn out.

[BGSWebDesign]

Hi Dean,

Any ideas on this would be helpful.  I'm back to the same problem, the number of Bounces is ZERO???  And the number of removes is ONE on a list of 10,000, it doesn't make sense?

Ok, just had a double check on a replied to Email, it seems the Reply to is going to my normal Email, where should the Reply point to when using the Bounce Email account?

I'm going to send the next batch with the Throttle Code again to see if it makes a difference, it could on the Removes, but I don't see why I'm getting ZERO Bounces, it appears the bounce handler is not working correctly, I changed the bounce Email password in the settings to a new one, did that affect it?

Thanks,

[BGSWebDesign]

Hi,

This is just a confirmation and for documenting that the Throttle code is REQUIRED for me to get the Bounce handling and the Removes (removals) to work properly.

I just now validated it with another list of 10,000.  That list now shows 229 removals and 130 bounces, a lot more like what I would expect.  But that sending was sent with the Throttle code (displayed above).

So, it appears the throttle code is doing something on my server setup to allow bouncing and removals to work.  

If anyone notices anything different please your findings as a reply to this message so I can get an idea of what may be going on.

Thanks,