Queue stuck

[DW]

Hi Brett,

The section of SMTP log you posted doesn't suggest a problem.

I am unfortunately out of guesses and unable to determine why zero byte files are being written without further hands-on access. Please create a support request. If absolute privacy is desired you could set up an extra LMP installation and FTP account where the issue can be recreated for me to work on.

Regards

[BGSWebDesign]

Hi DW,

Thanks for letting me know.  I've found what I think may have caused some of the problem. My mail queue was overloaded with 'Delayed sending messages', actually it was filling up near the top of my capacity setting for my 'bounced email address' mailbox.  I've had that cleared to zero, and wiped everything else, and now I'm not getting a stuck queue anymore.

I am still however getting the zero byte files, every time Resume runs.   I'll let you know if that becomes a problem. 

It appears that Resume.php is creating zero byte logfiles, however normal sending from within LMP works fine, and creates a normal logfile.

[BGSWebDesign]

DW,

Can you please tell me the mechanism (code) that sets the lm_sendp (queue protection table) completed field (lm_sendp->completed) to '0', so that it can start sending?  Is that done in Dailymail.php?

The reason I'm asking is that I think I found my problem.... I know it makes no sense, but none of my Dailymail followups send, BUT, every time I do a Selection Send (even to the same email addresses), it works just fine, so I suspected there was some other problem.

What I found is that my lm_sendp table had all of my Dailymail followup batches listed in it with completed field set to 'q', which meant that it would NEVER try to send those, even though they had just been loaded up?

Why that is occurring I don't know, so I'm trying to locate the code that should set that value (lm_sendp->completed) to '0' (ok to send) so that my Dailymail followups can continue to send properly? 

If I can locate that code, and make sure it is being run - after the queue of followups is loaded, then it's likely I can get everything running properly again.

[BGSWebDesign]

Hi,

One last piece of the puzzle. For some unknown reason my Listmail installation has stopped working normally.  What I've found is that the queue protection table lm_sendp gets loaded with the batch when Dailymail runs every day, and the 'completed' field ( lm_sendp->completed ) always gets initialized to 'q', which prevents the resume function from ever sending that mail, until apparently it gets set to '0'? 

Here is the code in Admin (line 2831) that sets it to 'q':

Code: [Select]

mysql_query("insert into $ptable values('','".addslashes($bat)."','".addslashes($qtype)."','".addslashes($doformid)."','$now','$now','','q');") or die(mysql_error());

I'm trying to locate the of code that sets the value of lm_sendp->completed to '0' so the sending can start?

Can anyone let me know where that code is - or if there is some other way that this value in lm_sendp gets set to '0' so that the sending can start?  I've noticed if I manually set that to '0' everything is fine and the sending commences, but if I do not, all that happens is the queue gets loaded with all the Followups and other Dailymail mail, but never gets sent - it just sits there?

I'm not sure why this just started happening, but my guess is it might have something to do with PHP code running differently?  I cannot think of anything else that would cause it, I've been running ListmailPro just fine for 12 years now!

Thanks in advance...

[DW]

An SMTP log of zero bytes or a queue that won't send could be because the queuing script dies or is killed by the server before sending.

A script like this could be used to see if your server ignores set_time_limit:

Code: [Select]

<?phpset_time_limit(0); // unlimited$time_to_run = 400; // seconds to run$start=time();$last=$start;while(1){ $diff=$now-$last; $now=time(); if($diff>0){  $last=$now;  echo ($now-$start).'.. ';  flush(); } if($now-$start>=$time_to_run) break;}?>

A crude band-aid that would at least see that messages that were able to be queued are sent out would be a script like this:

Code: [Select]

<?php// fix LMP queues that stall. include this script in resume.php after config.php and admin.php, or run manually from separate cron job. script requires writable queuefix.dat in same folderinclude_once('./config.php');include_once('./admin.php');if($file_version>'1.88') exit("Fatal Error: queuefix at http".(!empty($_SERVER['HTTPS'])?"s":"")."://".$_SERVER['SERVER_NAME'].$_SERVER['REQUEST_URI']." needs to be updated to use mysqli");$log='';$echolog=false; // set to true for output// read last recorded data$fdata=file_get_contents('./queuefix.dat');if($fdata) $log .= "existing data:\n".nl2br($fdata)."\n\n"; else $log .= "no existing data\n";$lines=explode("\n",$fdata);$data=array();if(!empty($lines)){	foreach($lines as $line){		list($batid,$msgcnt)=explode(':',$line);		$data[$batid]=$msgcnt;	}}// read current data$out='';$now=time();// v1.88 mysql_if($file_version<='1.88'){	$rows=mysql_query("select batid from lm_sendp where completed = 'q';");	if(mysql_num_rows($rows)>0){		$log .= "found queued messages, processing..\n";		while(list($batid)=mysql_fetch_row($rows)){			$log .= "batid $batid is queuing.. checking.. ";			list($count)=mysql_fetch_row(mysql_query("select count(*) from lm_sendq where bat = '$batid';"));			$log .= "count=$count\n";			if($count>0){				if(!isset($data[$batid])){ // no existing data, record it					$log .= "no existing data for batid $batid, recording:\n";					$log .= "out.=$batid:$count\\n\n";					$out.="$batid:$count\n";				} else {					// check if count is the same as last time, if it is fix the queue					if($count==$data[$batid]){						$log .= "count is same as last time, queuing stalled, running fix: ";						$log .= "update lm_sendp set completed = '0' where batid = '$batid';\n";						mysql_query("update lm_sendp set completed = '0' where batid = '$batid';");						$echolog=true;					} else {						$log .= "count changed, still queuing, ignore.\n";						// if count not same, still queueing up or down or now sending, so ignore it and start again					}				}			}		}	} else {		$log .= "no messages are queuing\n";	}// End v1.88} else {	// update for mysqli}// write out new dataif($out){	$log .= "writing out data:\n".nl2br($out)."\n";	file_put_contents('./queuefix.dat',trim($out));} elseif($fdata){	$log .= "clearing out data\n";	file_put_contents('./queuefix.dat','');}//file_put_contents('./queuefix.dat','test:test');if($echolog){	if(strstr($_SERVER['HTTP_USER_AGENT'],'Wget')===false) $log=nl2br($log);	echo $log;}?>

That script should also answer you question about the command to use to get such a stuck queue going.

Regards

[BGSWebDesign]

Hi DW,

Does this code: $file_version>1.88 indicate ListmailPRO version 1.88, or is that related to mysql?  I am running version 1.85 of ListmailPRO.

A few things.
1.) I do have the Zero Byte SMTP logs all the time, so I'm guessing the script is timing out, but why would it start doing that now when it's been running fine for 12 years?  I will run your test scripts and see
what I come up with.
2.) Thanks a million for the Band-Aid, is it better to run that by itself, or to put that code directly in resume.php? If I run it directly with a cronjob, can I run it every 5 minutes, or so? It appears it would do what I'm doing manually, checking that there is a 'q' in completed, change it to '0' and let that queue start sending. 

I know that once I set completed to '0', the emails go out, as I can see them sending out as the queue size goes down as sending proceeds - are you suggesting that those emails may not be sending out - if the script times out? 

That's a huge enlightment, as that would mean that anything I'm sending out, even large broadcasts, that I send to 70,000 or more would NOT be going out if the script times out, is that correct? The reason I ask, is that last week I've sent out several times to over 70,000 and I've seen a few times, where I did actually get the 'last screen' showing that ALL Messages sent, how fast they sent, etc...  in other words, it appears those very large sends were not timing out, would that be correct?  While often I will see the screen that says SMTP error, but then the Resume script keeps them going out... at least that's what I thought.

I will post here next with the results from the timeout script, to let you know what that shows.

Thanks a ton!

[BGSWebDesign]

Hi DW,

See my previous reply...

Here are the results of the Timeout script, does this tell you anything?
--
1.. 2.. 3.. 4.. 5.. 6.. 7.. 8.. 9.. 10.. 11.. 12.. 13.. 14.. 15.. 16.. 17.. 18.. 19.. 20.. 21.. 22.. 23.. 24.. 25.. 26.. 27.. 28.. 29.. 30.. 31.. 32.. 33.. 34.. 35.. 36.. 37.. 38.. 39.. 40.. 41.. 42.. 43.. 44.. 45.. 46.. 47.. 48.. 49.. 50.. 51.. 52.. 53.. 54.. 55.. 56.. 57.. 58.. 59.. 60.. 61.. 62.. 63.. 64.. 65.. 66.. 67.. 68.. 69.. 70.. 71.. 72.. 73.. 74.. 75.. 76.. 77.. 78.. 79.. 80.. 81.. 82.. 83.. 84.. 85.. 86.. 87.. 88.. 89.. 90.. 91.. 92.. 93.. 94.. 95.. 96.. 97.. 98.. 99.. 100.. 101.. 102.. 103.. 104.. 105.. 106.. 107.. 108.. 109.. 110.. 111.. 112.. 113.. 114.. 115.. 116.. 117.. 118.. 119.. 120.. 121.. 122.. 123.. 124.. 125.. 126.. 127.. 128.. 129.. 130.. 131.. 132.. 133.. 134.. 135.. 136.. 137.. 138.. 139.. 140.. 141.. 142.. 143.. 144.. 145.. 146.. 147.. 148.. 149.. 150.. 151.. 152.. 153.. 154.. 155.. 156.. 157.. 158.. 159.. 160.. 161.. 162.. 163.. 164.. 165.. 166.. 167.. 168.. 169.. 170.. 171.. 172.. 173.. 174.. 175.. 176.. 177.. 178.. 179.. 180.. 181.. 182.. 183.. 184.. 185.. 186.. 187.. 188.. 189.. 190.. 191.. 192.. 193.. 194.. 195.. 196.. 197.. 198.. 199.. 200.. 201.. 202.. 203.. 204.. 205.. 206.. 207.. 208.. 209.. 210.. 211.. 212.. 213.. 214.. 215.. 216.. 217.. 218.. 219.. 220.. 221.. 222.. 223.. 224.. 225.. 226.. 227.. 228.. 229.. 230.. 231.. 232.. 233.. 234.. 235.. 236.. 237.. 238.. 239.. 240.. 241.. 242.. 243.. 244.. 245.. 246.. 247.. 248.. 249.. 250.. 251.. 252.. 253.. 254.. 255.. 256.. 257.. 258.. 259.. 260.. 261.. 262.. 263.. 264.. 265.. 266.. 267.. 268.. 269.. 270.. 271.. 272.. 273.. 274.. 275.. 276.. 277.. 278.. 279.. 280.. 281.. 282.. 283.. 284.. 285.. 286.. 287.. 288.. 289.. 290.. 291.. 292.. 293.. 294.. 295.. 296.. 297.. 298.. 299.. 300.. 301.. 302.. 303.. 304.. 305.. 306.. 307.. 308.. 309.. 310.. 311.. 312.. 313.. 314.. 315.. 316.. 317.. 318.. 319.. 320.. 321.. 322.. 323.. 324.. 325.. 326.. 327.. 328.. 329.. 330.. 331.. 332.. 333.. 334.. 335.. 336.. 337.. 338.. 339.. 340.. 341.. 342.. 343.. 344.. 345.. 346.. 347.. 348.. 349.. 350.. 351.. 352.. 353.. 354.. 355.. 356.. 357.. 358.. 359.. 360.. 361.. 362.. 363.. 364.. 365.. 366.. 367.. 368.. 369.. 370.. 371.. 372.. 373.. 374.. 375.. 376.. 377.. 378.. 379.. 380.. 381.. 382.. 383.. 384.. 385.. 386.. 387.. 388.. 389.. 390.. 391.. 392.. 393.. 394.. 395.. 396.. 397.. 398.. 399..
--

I will also run the other script queuestuck next.

[BGSWebDesign]

DW,

Thank you for that code to unstick my Queue, I'm still not sure why it happens (queue sticks every day), after years of running just fine, but your code runs perfectly and gets things running again!

Most likely then, is it because the script is timing out, is that the problem?