*/15 * * * * /usr/bin/wget -O /dev/null -T 0 http://example.com/mail/resume.php?pw=YourDailyMailPass wget: timeout: Invalid time period `O'*/15 * * * * /usr/bin/wget -O /dev/null http://example.com/mail/resume.php?pw=YourDailyMailPass
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
[ <=> ] 42 --.--K/s
00:49:56 (410.16 KB/s) - `/dev/null' saved [42]
TTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
[<=> ] 0 --.--K/s a [ <=> ] 42 --.--K/s
batid=cab744, qtype=1, formid=2006041401253, started=2006-04-14 01:25:43, lastact=2006-04-14 01:26:09, report=(empty), completed=1
bat=e18c3a, battype=2, mtype=2, mid=85 the user ids are there for the records. This one is stalled... wget: timeout: Invalid time period `O'
00:49:56 (410.16 KB/s) - `/dev/null' saved [42]
I'm going to reset the completed to '0' to see if it will re-start (resume).
Here is what I found in lm_sendp, 2 entries, 1 is for DailyMail Report, the other is my batch which reads:
What exactly is going on here DW? It looks like the queue was loaded at 1:25:43, and then finished at 1:26:09, is that right?
Seems to be a typo. The time period should be set to 0 (Zero) not the letter O.
id=5, batid=7afbbf, qtype=2, formid=, started=2006-04-17 04:15:15, lastact=2006-04-17 09:29:27, report=Dailymail Report for 2006-04-17 04:15:15Totals: ..., completed=0
This mailing appears to be in the process of sending normally. It has responded within 1 minute How does it respond, I don't see that the numbers in 'Mail Sent' are going down, so how is it responding - and there is no record in the lm_sendp table?I notice that lastact is getting updated frequently - as if email is being sent out - BUT - the 'Mail Sent' counter is NOT updating in the header of LMP?
Ok, this time I deleted the top record in lm_sendq - since usually when it chokes it's because the top record is bad (bad address/or bad deleted record) and NOW the Mail Sent is being updated, it's running again
Does the 'resume' automatically updated the records in the queue and delete them as they are sent out - does it updated the 'Messages Left' displayed in the header as I update my LMP page?
Yes, it's starting to make some sense. If ListMail were to be caught in a loop on a single user it would mean no other rows are processed and deleted from the queue table.
cprk.com.myI also recommend enabling the $smtp_debug var in admin.php so we can see what's happening with SMTP.
Instead of covering up the issue I would really like to get to the bottom of it. I will try some tests with invalid addresses, but I fear this problem may only be producable on a server running particular mail server software.
I think that yes, Brett, you should enable the SMTP log. It's possible that an SMTP response is not being interpreted properly, causing a loop.
Apr 18 11:08:02 serv2 qmail: 1145383682.898725 delivery 16100729: failure: 219.94.65.171_does_not_like_recipient./
Remote_host_said:_550_<test1@cprk.com.my>:_Recipient_address_rejected:_User_unknown_in_virtual_alias_table/Giving_up_on_219.94.65.171./
Apr 18 11:11:59 serv2 qmail: 1145383919.164225 delivery 16100735: deferral: Sorry,_I_couldn't_find_any_host_by_that_name._(#4.1.2)/
If you give me the exact specs for your custom script (ie. scan queue table every 1 minute while mailings are active within 1 minute) I'll set it up for you then do more research on this.
SMTP Server (recommended):
settings - port: 25, reconnect every 249 emails.Hi Mike,
No, I'm not running sendmail, I'm running LMP own SMTP
mailer, from LMP Configuration page:Code: [Select]SMTP Server (recommended):
settings - port: 25, reconnect every 249 emails.
Let me know if you have any other ideas - this thing seems to be stalling constantly - about every 30-45 minutes!
Is the 'reconnect every 249 emails' too low? I used to have it at 499, but lowered it when I started having problems.
This really sounds like a broken SMTP, but it's so hard to say since you say you weren't having this problem in V1.84, what about V1.85? If so maybe Dean changed something in the SMTP routines that broke for your mail program.
This is why bounce.cgi is recommended for large lists.
so that means there's a problem with LMP in this situation - it may work with bounce.cgi - but it's impossible doing it the way I do - if resume.php will NOT run the bounce processing!
I absolutely can NOT lose sales when bounce.cgi is running every .5 to 1 second and sometimes more when large lists are mailed.
Bounce processing is never activated by resume.php - I hadn't thought about it being necessary.
I totally understand. What you can try is having your web host enable a higher 'maximum # of connections' to the web server. Also, some MySQL optimizations can likely be made. If this doesn't help you may want to consider a dedicated server - I know one user running a high-powered dual-processor server with 1 million addresses using bounce.cgi.
Regarding bounce.cgi - if you can show me with good testing/results that it's possible to run large lists such as I mention and still be able to process link tracking requests - I may consider switching - but until then - I'll need to find a way to do it on my own - because I absolutely can NOT lose sales when bounce.cgi is running every .5 to 1 second and sometimes more when large lists are mailed.
Brett, I honestly believe in this case it is almost hopeless to get at that POP mailbox and the longer you let it go the worse it will get. I'd cut my loses and overwrite it now to start fresh.
Brett, if you can wait a little longer I might be able to solve the problem and avoid the need for the custom hack I offered.
This is good information. I do not believe this is as simple as "email addresses with two periods", however. I did some tests and it seems that all domains can have quick or delayed DNS replies. This is probably due to nameserver response time.
This must be the problem. I will do some tests with the access info you provided and get back to you soon
test5@dsds.com obviously a bad address - but LMP chokes on it, I don't remember v. 1.84 doing that - I must have modified the code, but have looked for over an hour this AM for it and can't find what I changed - there is one thing with the bounce processing that was different but I don't think that is it...$cmd = "select id,list,bounces from $utable where email like '".addslashes($email)."'";
$urows = mysql_query($cmd) or die('admin-45-'.mysql_error());pipelining support enabled.
NOOPmsg=250 2.0.0 OK
PIPELINE-FROMmsg=250 2.1.0 ... Sender ok .
RCPT TO: <>
RCPTmsg=553 5.0.0 <>... User address required
DATAmsg=503 5.0.0 Need RCPT (recipient) .how do you get someone entering a dummy address and it being added to a list? If they do not exist they won't get confirmed. Are these not dropped after not being confirmed?
how do you get someone entering a dummy address and it being added to a list?
On Dean's server, which have access to, I'm getting a new type of error "lost connection to MySQL
"Lost connection to MySQL server during query"
I've found that this error also will occur if for whatever reason, the client takes longer than <connect_timeout> seconds to connect to the server. The default value is 5 seconds. You can increase this value in your my.cnf file or elsewhere, though it would be ideal to figure out why it's taking so long to connect.
admin-26-Lost connection to MySQL server during query
[mysql]
set-variable = connect_timeout=15
My host says they have the time out set to 10 but it has been that way for 6 months. The 1.84 version is where this started.
Also, the domains that are causing the problem ARE failing dns look-up (not just at the host) - so they are the problem.
Can't you just "if(timeout) skip" ???
I will see if I can place some sort of timer on the read function so you can set how long to wait for response to these requests.
if($lastmsg == '503'){
fputs($ssock, "RSET\r\n");
$rsrvmsg = fgets($ssock,1024);
$errmsg .= $srvmsg;
if($bugs){ echo "RSETmsg=$srvmsg<br>"; flush(); }
flush();
$skiptonext = 1;
$reconnect = 1;
if($bugs){ echo "Bad RCPT response ($email), skipping.<br><br>"; flush(); }
} else {
$lastmsg = substr($srvmsg, 0, 3);
if ($lastmsg <> "250"){
$skiptonext = 1;
$reconnect = 1;
Skipping may be possible but it won't be easy.Why? I don't see what's so hard about it, really, and I didn't write this code, you did... Is there something I'm not thinking of, as from what I can tell it only involves a simple reccount() check in the send loop (as I mention above), if it stays the same for several minutes, obviously there's a problem with sending... so delete the top record in the table lm_sendq - can you fill me in DW on anything else?
function getsmtpmsg($xsock){
global $smtp_timeout;
if(!is_numeric($smtp_timeout)) $smtp_timeout = 9;
$data = "";
stream_set_timeout($xsock,$smtp_timeout);
while($str = fgets($xsock,1024)){
$data .= $str;
if(substr($str,3,1) == " ") break;
}
return $data;
}Ok, it's been 10 days since I first posted the first thread - when are we going to get some code to fix this issue? btw do you have a decent test bed DW? It doesn't sound that way if you're not able to duplicate these types of hiccups?
$smtp_timeout = 4;I plan to include a tool to weed out these non-existent addresses before they are emailed ASAP.
while($str = fgets($xsock,1024)){?It might be better to run the verification after import rather than on-the-fly... either way it should work.
The question is, do we want to do this on EVERY mailing for EVERY message and ultimately take ALL unnecessary load off of the SMTP server and bounce routines? Sounds like a good idea to me.
While I'm at it, why even send to an SMTP server at all... I could include a new mailing option for ListMail to be it's "own" SMTP server.
It seems that adding this new function (or adding the delay) to the smtp calls is helping get the mail delivered but it SLOWED down my emails tremendously, it's taking 2-3 times longer to get my mail sent out!
if it does NOT change in 30-40 seconds, delete the top record, instead of throwing this LONG DELAY into every single SMTP call
If you get several of these in a block of 50 you might notice your counter incrementing very slowly. What we need to do is clean your list with a script such as mentioned here to remove these failing addresses from mailing entirely
As I stated earlier, it would be MUCH BETTER (ie FASTER and SIMPLER) to just place a simple command in resume.php, watch the reccount() on lm_sendq and if it is STUCK ON ONE RECORD, delete that record and move on - there can't be that many...
if(!is_numeric($smtp_timeout)) $smtp_timeout = 9;if(!$smtp_timeout || !is_numeric($smtp_timeout)) $smtp_timeout = 9;Please enable the SMTP log either globally or when using "Send Email" and you should be able to see exactly how many delayed/invalid addresses there are.
One problem with this is that the lm_sendq table is not sorted. That is, it has no 'auto-incrementing id' id which can be used to sort processing. The first email one 'resume' may not be the first email the next. This was done to prevent having to update the database index every time rows are added or deleted, which causes overhead.
The call to stream_set_timeout appears correct as per the PHP documentation
I have found it required to add
"stream_set_blocking($fp, FALSE )"
prior to any fgets(), fread(), etc. to prevent the code from hanging up when remote files are called and the response is slow.
How would this be any different than what I do? It's not, I just go into lm_sendq and delete the first record, that's it, the mailing continues on from there UNTIL it hits another bad email address - I don't see where this is a problem?
look at the 'Ridera' test code posted up on Php.net
I have found it required to add
"stream_set_blocking($fp, FALSE )"
prior to any fgets(), fread(), etc. to prevent the code from hanging up when remote files are called and the response is slow.
How would this be any different than what I do? It's not, I just go into lm_sendq and delete the first record, that's it, the mailing continues on from there UNTIL it hits another bad email address - I don't see where this is a problem?
I may be mistaken but without an auto-incrementing ID we can't be sure of the order of the returned rows during sending, resuming, or with a manual 'Browse' as you do.
SELECT * FROM `lm_sendq` WHERE 1, if you do the exact same query (or similar) in LMP - YOU ARE GOING TO GET THE EXACT SAME RECORD!I understand that you seem to be able to grab the top row consistently, but I don't think it's guaranteed.
This is done because an index takes time to update when inserting or deleting rows to maintain quick sorted results, so it would slow down the queuing and sending processes.
If you want a custom script to do a general select and delete of the top row (if the # of entries hasn't changed in X time) I'd still be willing to program it for you.
I had a look (at Ridera code on PHP.net) and it appears to be written to manually time the connection with blocking set to false. Blocking off means the read command doesn't wait for data from the server.
ListMail *could* do it this way, but I wonder if it would be more processor-intensive. Look at all the function calls, variable comparisons, etc. required.
I have found it required to add
"stream_set_blocking($fp, FALSE )"
This could be applicable here and would mean ridera's solution is the only solution. I will have to investigate this further possibly by inducing delays etc. to simulate what you experience on your server. It will be useful to see what's happening in your SMTP log. While it won't tell us how 'slow' the mailing goes, it will at least rule out the possibility of the majority of delays coming from a very large number of bad addresses / reconnects.
Regards
, if you do the exact same query (or similar) in LMP - YOU ARE GOING TO GET THE EXACT SAME RECORD!
anyone in their right mind would never write a DBMS that would pull different records every time a query is done on an unindexed table, that's pure insanity....
This is what I've asked for, for 2-3 weeks now.... of course that's what I want - NOT ONLY THAT, it is THE BEST WAY to solve this problem - and NOT by throwing in 12 second SMTP delays - you've slowed my mailing queue sending tremendously going out SMTP, the solution you have is NOT going to work
The change to getsmtpmsg has slowed down my sending considerably - is that supposed to happen DW? I tried it with $smtp_timeout=9, it's slow, so I moved it to $smtp_timeout=4 as you suggested and it's even slower, so now I'm going the other way $smtp_timeout=12 - WHAT am I affecting here, it seems like it's stalling about every 30-40 emails now???
getsmtpmsg() took 0.00039899999999998 ms
getsmtpmsg() took 0.00037000000000004 ms
getsmtpmsg() took 0.002756 ms
getsmtpmsg() took 0.005205 msWhat I'm wondering about is if this new function would be somehow NOT getting some of the mail through because of this:Code: [Select]while($str = fgets($xsock,1024)){
Depending on whether your server mysql_timeout is the default 5 or raised to 10 the proper/safe setting should be 4 or 9. You could go lower, to 2 or 3... if your server's DNS is performing well most lookups should take less than 1 second.
I believe this is the reason your mailing process ultimately "stops" and does not continue past bad addresses, no matter how long they happen to take.quote]
Not sure what you mean here - 'stops'? My mailing works fine, it USED to Stall - BEFORE I added the smtp_timeout code into the function you posted, but now it does NOT STALL - sometimes I will get the Server Error, Server Said (WITH NOTHING for the Server Error Message).
I get the feeling it's running slower because, like I said, it seems to take 2-3 times as long to do a mailing - NOW what I believe is happening is this is because it is STALLING at the bad email addresses, and then has to wait 5 minutes for the resume.php to PICK it up again and continue, if there are enough bad email addresses this could slow things considerably, correct?QuoteWe can only set a cron job to run once a minute. If
Why, can't it be included in resume.php - oh I see, you want to hit it more frequently - probably a good idea, but as you see above things seem to be working just fine, I'll get you the SMTP logs and you can see, probably what I should do is makeup a test list with all those BAD email addresses and see how those logs look?
Keep me posted on what you do to the code, as I said, I'm running SMTP_timeout at 12, so I think you're going to need to make this a global configuration option...
UPDATE
=====
Here's some Bad email addresses I tested, this is what happened, started test, 4 bad email addresses, 2 of my own, first address, chokes, gives me message: Sending error. Check your mail settings. (Server Said: ) - blank nothing in 'Server Said', click on Go To Config, see 4 messages left... click on Link Tracking, 1 minute later, ALL messages are done, I've got a SMTP log for it too... I'll send you that as well...
It's at 12 as I said... when I had it 9 things were slow, when I tried it at 4 things were even slower, at 12 everything seems to run fine?
QuoteI believe this is the reason your mailing process ultimately "stops" and does not continue past bad addresses, no matter how long they happen to take.
Not sure what you mean here - 'stops'? My mailing works fine, it USED to Stall - BEFORE I added the smtp_timeout code into the function you posted, but now it does NOT STALL - sometimes I will get the Server Error, Server Said (WITH NOTHING for the Server Error Message).
I get the feeling it's running slower because, like I said, it seems to take 2-3 times as long to do a mailing - NOW what I believe is happening is this is because it is STALLING at the bad email addresses, and then has to wait 5 minutes for the resume.php to PICK it up again and continue, if there are enough bad email addresses this could slow things considerably, correct?
<?php/********************check lm_sendq every minute, if entries check if changed and delete top rowauto-resume will then pick up the remainder of messages when the mailing has failed for one minuteplace this script in your listmail folder and set up a cron task for every minuteie: * * * * * /usr/bin/wget -O - http://example.com/mail/autodelete.php?pw=1234A new table and row are required to store the last count:SQL: CREATE TABLE lm_sendq_num (lastcount MEDIUMINT UNSIGNED NOT NULL); INSERT INTO lm_sendq_num VALUES ('0');********************/include('./config.php');include('./admin.php');if($_GET['pw']<>'1234') exit('access denied: wrong password');$email_notify = 1;$email_to_notify = 'test@listmailpro.com';$tbl = 'lm_sendq_num';$DEBUG = 1;// leave this$msg = '';// retrieve countslist($oldcnt)=mysql_fetch_row(mysql_query("select lastcount from lm_sendq_num;"));list($cnt)=mysql_fetch_row(mysql_query("select count(*) from lm_sendq;"));if($DEBUG) $msg .= "cnt=$cnt, oldcnt=$oldcnt\n";if($cnt==0){ // if no items in queue reset the oldcount if($DEBUG) $msg .= "no items in queue - resetting oldcount and exiting\n"; mysql_query("update $tbl set lastcount = '0' where 1;");} else { if($DEBUG) $msg .= "$cnt items queued, checking oldcount\noldcnt = $oldcnt\n"; // items in queue, check the oldcount // if no oldcount update to new count if($oldcnt==0){ if($DEBUG) $msg .= "oldcnt is 0, update to new count ($cnt)\n"; mysql_query("update $tbl set lastcount = '$cnt' where 1;"); } elseif($cnt==$oldcnt){ // otherwise, check if its the same and delete the first row if so if($DEBUG) $msg .= "queued items = lastcount, delete top row\n"; // get data list($id,$bat,$battype,$mtype,$uid,$mid,$xtra)=mysql_fetch_row(mysql_query("select * from lm_sendq limit 1")); // delete row mysql_query("delete from lm_sendq where id = '$id';"); list($uem)=mysql_fetch_row(mysql_query("select email from lm_users where id = '$uid';")); $af = mysql_affected_rows(); if($af>0){ $ncnt = $cnt-$af; mysql_query("update $tbl set lastcount = '$ncnt' where 1;"); $msg .= "The following row was deleted from the lm_sendq table:\n\nid=$id bat=$bat battype=$battype mtype=$mtype uid=$uid mid=$mid xtra=$xtra\n\nThe user has the email: $uem\n"; } else $msg .= "Error, row not deleted."; // notify if($email_notify) mail($email_to_notify,'LM Auto-Delete',$msg,"From: \"LM Auto-Delete\" <noreply@example.com>"); }}if($DEBUG){ echo "msg=".nl2br($msg); if($email_notify) mail($email_to_notify,'LM Auto-Delete (Debug)',$msg,"From: \"LM Auto-Delete\" <noreply@example.com>");}?>