At work on v1.85

[don1]

Hey Dean...

I'm going to put a few more features in and then will be releasing. It should be ready in the next week  or two  or maybe three.  

I'm ready to take a look at the new version...  Can't wait  

[DW]

Brett,

Have you had any problems with cron-style followups lately?

Don,

I am working away on the update and my new site @ lmhost2.com (I changed the front page sales copy - let me know what you think).  I've been a little distracted by a lot of support.  I will put some emphasis on the ListMail update tonight and over the coming days - there are a lot of great suggestions to implement!  

Regards

[don1]

Dean,

The new site is looking good..."it's getting better all the time" (I just got home from seeing Paul McCartney)

Anyway...I did not want to bring this up (but I will) one comment about LMP...I love the program and can't wait for the new updates...  I personally do not think it will be peoples first choice for an autoresponder or online newsletter etc mainly because there is no WYSIWYG html editor incorporated into the program.

I do not need this feature but I think the average user is "expecting" it in the program.  Most of the other similar programs do have some sort of WYSIWYG html capabilities.

And one more thing, like I said before.  I think it would be a good a idea to release smaller updates more often..."Less is more"..

I want LMP to be the number selling program of it's kind...I think it will benefit all of us ...Good luck, and I look forward to the next release.

Don

[BGSWebDesign]

Hi DW,

Ok, I've installed two new .php files, Dailymail.php and Admin.php (for testing purposes only, to see what's causing the sql query problem).

One thing I noticed, I belive you should put this in Dailymail.php (line 23) - when running tests from the Command line? Of course in normal mode, they then need to be commented out - so PERHAPS, you should consider this - add a 'Testing' checkbox on Config Screen, if that's ON then all of these vars are set to 1, otherwise, they are set to zero:

Code: [Select]

$nomaint = 1;
$nosched = 1;
$nofups = 1;
$nobounce = 0;


right after this line:

Code: [Select]

if($pw) $isadmin = checkpw($pw);

since I am running from the Command Line, I'm not sure those vars get set in the html code, so I added the code above, to ALLOW bounce processing, and turn off all other processing...

Adding the code you mentioned:

Code: [Select]

$cmd = "select id,list,bounces from $utable where email like '".addslashes($email)."'";
echo "CMD=$cmd<br>";
$urows = mysql_query($cmd) or die('admin-45-'.mysql_error());


Did you try adding "addslashes()" around the $email variable as I mentioned?  If you did, you may not have had a chance to test it. Before doing that I'd be curious to see the output of the query causing the error.

I did not try it WITHOUT the addslashes, but I can tell you adding the addslashes solved the problem, as the bounce processing went through the bounces as expected and completed it's execution as normal... so that fixed it!     Please update your code accordingly, and add the Addslashes code in Admin.php (line 2489).

I have finished work on the new dailymail options.  You can now disable individual parts of dailymail when running it manually.  I have sent the new files to you by email... quite certain I've got it all sorted but please let me know if you notice any errors.

Yes, it works fine, thanks.... except see note above, I believe you should add the var setups at the beginning (for testing purposes only), and ALSO, you should consider writing to a .log file (for running from browser) - then write the .txt file out to browser, I don't see ANYTHING on the screen when running this from the browser, so generally I run from command line to see what's going on...

Have your cron-style followups been ok the last few days?

Yes, all has been fine with the followups, looks like you're good for the update.... can't wait to see what else you've added...

[DW]

Brett,

Glad to hear you braved the new code.

Adding the addslashes solved the problem, as the bounce processing went through the bounces as expected and completed it's execution as normal... so that fixed it! Please update your code accordingly, and add the Addslashes code in Admin.php (line 2489).

I added the following to the top of bounce() function in admin.php to prevent unnecessary wasting of resources.

Code: [Select]

if(!valid_email($email)) return false;
addslashes() is still beneficial to defeat sql-injection if the valid_email() regexp is defeated.

That was a pretty big hole... :oops: I need to get an update out ASAP.

since I am running from the Command Line, I'm not sure those vars get set in the html code, so I added the code above, to ALLOW bounce processing, and turn off all other processing...

No, the vars don't get set when running from the command-line, so everything executes.  Your addition is one way of working around it or you could have alternately set the variables via GET in the URL.  ie. http://example.com/mail/dailymail.php?pw=mypass&nofups=1

ListMail automatically grabs all POST and GET variables regardless of the "Set Globals" PHP option. (top of admin.php)  Note that POST overrides GET.

I will consider adding a "test mode" for all dailymail operations - good suggestion.

I don't see ANYTHING on the screen when running this from the browser, so generally I run from command line to see what's going on...

The variable that determines display is on this line of dailymail.php:

Code: [Select]

if($warn=='no') $outp = 1;
I don't appear to be clearing the $outp variable before or after this if() statement, so you could try adding &outp=1 to the URL.

Thanks a lot for your recent testimonial, by the way (where's your site link??)!  I often feel at odds with you, especially when I make mistakes!  Indeed, I carry a great deal of responsibility for a number of internet businesses' successful operation.  As you can imagine, when things go wrong it can be a bit stressful...  Let's just say having your post there means a lot to me.  I also greatly value your input and help developing ListMail, and your patience when things get busy.  Thanks for being such a great client.

I'm at work on more features for the update.

Regards

[BGSWebDesign]

Hi DW,

I added the following to the top of bounce() function in admin.php to prevent unnecessary wasting of resources.

I'm not sure that's the way to handle that!  Think about it, since the bad email address MAY already be in the database, right?   If the bad email address has a chance for slipping into the database to start with, it does need to be REMOVED from the database with the bounce code, so that it's not there anymore?  I would prefer to lean toward the 'safe side' of things and make sure the code gets executed (by using addslashes) to ensure that AL BAD email addresses are removed from the database - this will guarantee that....

If there is any chance the bad email address may be in the database, then using the 'addslashes' code would make sure that it also gets removed from the database, but only if there is NO chance for the bad email address to make it into the database would the code you mention work you must know the answer to that. but please re-consider as  I mentioned to guarantee that any and ALL bad email addresses are truly bounced and removed from the database.... the last thing I ever want is bad email addresses hanging around in the database...  I would prefer that you leave OUT THE "valid_email" code, and instead just put the addslashes code in...

I will consider adding a "test mode" for all dailymail operations - good suggestion.

Thanks, that will solve many problems!

One thing, I forgot again to add my 'throttle code' into admin.php which has cost me many lost emails (and sales)... I suggest again that you consider adding that code, and adding a SWITCH for it in Config, enable Throttle, and include options for setting the number of emails to queue, and the throttle delay time...  here's the code again:

Code: [Select]

if(($y % '100')==0 && $y <> sizeof($sendq)) sleep(25);
I place this at line 1712 in the current admin.php.

so you could try adding &outp=1 to the URL.

I don't see the output when running from the browser - so  you'll need to think about how you can fix that, IF you want the user to see the output...

I'm at work on more features for the update.

I can't wait to see the new ones.... thanks again for your great work!

[DW]

Brett,

Think about it, since the bad email address MAY already be in the database, right?

Actually no, there shouldn't be any.  Since a very early version ListMail has done email format checking on all imports, additions, and edits.  The only time this is ever an issue anymore is if a client is using a 3rd party script to add users to the ListMail database directly.  It might be best to implement a feature that could be run every so often to verify email addresses.  Even over long periods of time, however, without using a 3rd party script I don't think you will get any bad ones.

One thing, I forgot again to add my 'throttle code' into admin.php which has cost me many lost emails (and sales)... I suggest again that you consider adding that code, and adding a SWITCH for it in Config, enable Throttle, and include options for setting the number of emails to queue, and the throttle delay time... here's the code again:

I'll see if I can get this feature into the update - it's a fairly easy one.  I'm not sure what you mean by setting the number of emails to queue, though - that could be more difficult.

Right now I'm diving into checkboxes on the User Database page as it's been far too long without them.  With them I'm adding mass user operations.  ie:

WITH [Selected] OR [All 123 Results] (spans multiple pages)
DO [Copy to List] -> [1: List Title] -> Next Followup[1][2][3]
OR [Move to List] -> [1: List Title] -> Next Followup[1][2][3]
OR [Remove]
OR [Remove From All]
OR [Create Selection]
OR [Set to Followup][1][2][3]
OR [Change Data] -> [Field] -> [New Value]
etc.

This should be fairly straightforward to implement and will be very useful to many, I'm sure.  Can you think of any other operations that could be put in here?  Do you think I should also add mass user operations for User Selections in light of the above feature(s)?

I will finish this and some changes to followup addition/renumbering I promised to Don before considering an official update.

UPDATE: After doing some thinking I probably shouldn't be working on the user checkboxes/mass user ops right now... While I did make some progress on it it is more time consuming than I had previously foreseen and way off track with some of the improvements I had planned on making ASAP.  I will finish what I promised, add some other very important things like queue/dailymail protections and a resume script, add a few easier features and get the release out.  If I manage to work quickly enough perhaps I will complete the above. If not, it'll come soon.

Unlimited custom fields may not make it as it is a difficult feature to implement, requiring many database / script changes, and I really want to get the new files out due to the bounce() security issue.  I may also quickly implement some simpler requested features such as your Copy List feature which I just need to find room for.

After this update there will still be lots I want to get done quickly.

Regards

[DW]

Next up is Don's request:

Being able to ADD followups BETWEEN existing followups.  As I'm going to start on this tonight I thought I'd try to get some feedback.

I've been doing some thinking about this one as it's a high priority.  It should be fairly quick to implement once I get it all planned out.  

A) Users on a followup BEFORE the one added proceed as normal, receiving the new followup in turn.

B) Users on a followup AFTER the one added proceed as normal and never receive the new message.

Some issues that I have not completely figured out follow:

1) Adding a new followup and giving it a lower existing seq # could create a JavaScript confirmation pop-up warning of the automatic re-arranging of users.  This is easy enough.

2) Changing an existing followup Seq # to a LOWER existing number should result in the same thing.

3) Changing an existing followup seq # to a HIGHER existing number should ALSO result in re-arrangement, with a gap left by the old moved # (automatically skipped).

4) This is a tough one.  What if you change the list of an existing followup to another list that contains the same seq #.  It should have the same warning and re-arranging...  This will take some JavaScript storing EACH list # and ALL of its seq #s to be checked upon saving.  This data would need to be loaded on both the followup editing / adding page and the main followup listing pages.  For those with thousands of followups across dozens of lists the page would load a bit slower (not too much, considering the data is all integers), but it should work perfectly.

4a) Update: This could also be accomplished with an intermediary 'warning' page eliminating the bulk of JavaScript and giving you even more options!

5) If the target seq # does not already exist do NO re-arranging.

6) To take out the guesswork when you're on the message editing page and can't see your list of followups I could also change the "Add Followup" button to something like this:
Add followup.. (Select Box)[At beginning of sequence][After 1][After 2][After 3][etc][At End of Sequence] (Submit) [GO]

Maybe I do have it figured out.  I'll stop talking and get to work... No responses necessary.

Regards

[DW]

I have made progress on the new followup feature.  At the same time I am attempting to eliminate 'gaps' and dupes in seq #s when possible.

NEW FEATURES (taken from new followup help page):
- On the Followups main page or when adding or editing a followup simply set the Seq # to an existing value, the one you want the message to be inserted before.  When you click Save you will be presented a warning box informing you that your existing users will be automatically re-arranged upon insertion of the followup.  When a followup is inserted all followups with a higher seq # increment by one along with their users.  Users who were previously at the end of sequence are placed at the new end of sequence with a delay of 0.

- To delete a followup click "Delete" next to the followup on the main followup page. All followups with a higher seq # decrement by one along with their users. When deleting the last followup in sequence all users found to be at or above the 'end of sequence' are adjusted to be at the new lower value - therefore, they will be in queue to receive the next followup when or if you add it.

UPGRADE:
- remove followups with duplicate seq #s and email data to admin
- update all users on non-existent followup to next followup or eos (end of sequence)
- clear out existing gaps. simple loop through followups on each list  incrementing a variable ($i) starting at 1.  if $i does not match the next seq # the seq # is changed to $i, along with all users with that seq #.  repeated until all followups are in order. all users at new eos or higher are placed at new eos

:?: Eliminating duplicate seq #s on upgrade is essential for the renumbering followups routine.  Both are not really required but are more cosmetic. I could take them out in favour of simply updating all users at eos or higher to the proper eos.  This might take some of the stress out of the upgrade on live installations.  I have tested the features quite thoroughly and believe I have created and compensated for all scenarios, but since we can't/aren't eliminating gaps and dupes completely it doesn't seem necessary.

NOTES:
Due to the severity of some of these operations I would release a Beta version in the hopes that some knowledgeable ListMail clients would help with testing.

This will not completely eliminate the ability to create duplicate seq #s, which currently serve no purpose and may/will cause undesired results.  You are able to create duplicate seq #s if you wish using the confirmation dialog.  It will also not completely eliminate potential gaps, such as when moving followups (though, I will be eliminating this one) or renumbering them manually.

I am not currently regularly eliminating gaps unrelated to the current insert/delete operation.  Should this be a regular process?

MORE FEATURES
I added a simple "Reset Users" button on the Followups page, but is it enough?  What about a "Reset all users on [Followup X or EOS] to Seq [ ] Delay [ ]" feature?  This should/will also be possible with the upcoming mass user operation features.

Let me know your thoughts

[BGSWebDesign]

Hi DW,

Your work sounds very promising, I like how you've described it and believe keeping the addition direct entry of sequence numbers is fine...

Regarding this:

What about a "Reset all users on [Followup X or EOS] to Seq [ ] Delay [ ]" feature?

That would be GREAT!  Is this something separate from LOOPING, or do you see this replacing the LOOPING feature?  I could see that this would be used to set a sequence to LOOP after a specific followup, and that is VERY MUCH needed!

remove followups with duplicate seq #s and email data to admin
- update all users on non-existent followup to next followup or eos (end of sequence)
- clear out existing gaps. simple loop through followups on each list incrementing a variable ($i) starting at 1

This all sounds promising, but since I have little trouble with sequence numbers/followups it's not very helpful to me, and it's not likely that I will test this for you.... as you know I'm busy testing the new CRON scheduling features which are working brilliantly  

Keep up the good work and I look forward to the updates coming more frequently, especially hoping for a 'COPY FOLLOWUP' option, or 'COPY GROUP OF FOLLOWUPS' from one list to another... MUCH NEEDED...

Background mailer (kick-start) and Split-Tested Mailings/Link Tracking are still also VERY IMPORTANT- see my original top 10 list posted on page 1 of this thread...

Really, though I believe you could solve ALL Of this by just posting the list in a Poll (you can do that in this forum?), and ask the users which are most important to them, that way you can get a feel for what is MOST important to MOST users - and this should be your priority, over anything else, even what others want you to do...  please consider doing that...

[BGSWebDesign]

Hi DW,

I've been using the CRON scheduled followups and they're working great, in fact, I LOVE THEM, but... there's still one issue that comes up, here it is:

I schedule followups in specific lists to run once a month, on the 1st, 2nd, 3rd, or 4th week of the month... I can do that easily by specifying the CRONs as follows:

Code: [Select]

FOLLOWUP LIST1:   1-5 * 2-4
FOLLOWUP LIST2:  8-12 * 2-4
FOLLOWUP LIST3: 15-19 * 2-4
FOLLOWUP LIST4: 22-26 * 2-4


As you can see these followups go out once a month for each of the corresponding lists, but only on Tuesday, Wednesday, Thursday, and only in the 1st week (for list1), 2nd week (for list2), etc...  NOW, here is where my question comes in...

Using this setup, I can do fine with covering all of the Months, even those that start midweek, and I can still hit my 4 weeks every month... BUT, I'm still left some months with an extra week (week 5) where I could make good use of a scheduled followup - IF AND ONLY IF there is a WEEK 5 in that month, if there is not, I'd prefer that followup to be SKIPPED... so you can see the problem here, IF I schedule a 5th week as follows:

Code: [Select]


FOLLOWUP LIST5: 28-31 * 2-3


That followup will ONLY HIT if there is a 5th week (that is, a day from 28th to 31st that falls on Tuesday-Wednesday), that is fine and seems to limit that followup to ONLY those months with a 5th week (according to my definition), but HOW can I advance this FOLLOWUP if it is NOT hit during the month?  If you're looking for an example see September 2006, there are 4 FULL WEEKS under my CRON Schedule, but the 5th week here is NOT available, since it comes too late in the month and is not a true 5th week.

I think what I'm after here is some type of WEEKLY CRON parameter (with a MONTHLY reset) that allows for ADVANCEMENT if it is NOT hit during that month?   I know that this is NOT part of the standard CRON setup, but it would be interesting to think about adding to your CRON class DW?   Unless you can think of something else, I can't see any way I can setup a followup to hit the 5th week - AND to also SKIP a month if it is not HIT?

What do you think DW?


Is there some way to force a followup reset - but it needs to be done automatically so I don't have to watch the calendar every month and see when it happens...

[DW]

It would be quite easy to program and schedule a script to run once a month, before dailymail on the 1st, resetting your users to the first followup.  Would this solve it?

Regards

[BGSWebDesign]

Hi DW,

It would be quite easy to program and schedule a script to run once a month, before dailymail on the 1st, resetting your users to the first followup. Would this solve it?

I think I should have said 'skipped' and not reset.... read below...

I don't believe it will solve it, since some users would need to be reset to different followups, the FOLLOWUP they need to be reset to is the one IMMEDIATELY AFTER the FOLLOWUP that would NOT get triggered at the end of the month (on 4 week months)....  as you see, this will not solve it..
Really what needs to happen is that FOLLOWUP needs to be skipped...

Have a look at a yearly calendar and you'll see what I mean... Most of the months have 5 weeks in them according to my schedule, only a few will NOT have 5 weeks, on those that do NOT have 5 weeks then this FOLLOWUP needs to be skipped, and the next following one go out INSTEAD, so that it skips that one on months that do NOT have 5 weeks in them (according to my schedule)....

I could post detailed info, but if you look at a calendar you'll see what I mean... this FOLLOWUP needs to be SKIPPED only on months that do NOT have 5 weeks in them, and the next followup goes out instead (the regular monthly followup)....  let me know if you need further details... but the message above and a calendar should show you what I mean (especially September 2006)...

[DW]

This won't necessarily be easy but should, of course, be possible..

It sounds like you don't want to reset because you have more emails in the sequence.

What about running a custom script before dailymail each day (or just the last few days of the month) that simply skips the message for users on that followup, if required.  We could use the same cron-style function, and the PHP date() function, in this custom script to verify when to skip users.  We could even check future dates if necessary...  How do we determine which months to run it and which months not to? :? This is a bit confusing.

Regards

[DW]

Let me know if you can figure out the logic behind the custom script you want, Brett.  I'm racking my brain with the next update..

I've been busy over the last 5 days or so working on LM.

Here's the latest [view Changes.txt] along with some notes:

- added storage of dailymail 'last ran' time. dailymail will not run via cron task until 23h55m from last ran.

This addition protects against runaway cron tasks that seem to 'refresh' the URL after a certain time.  The last execution time can optionally be updated (though it might/probably will cause ListMail to skip your next dailymail!) when running dailymail manually.

Note: This feature might be detrimental when testing whether or not the cron task is configured correctly.  For example, when testing 'wget' alternatives I like to run the cron task every minute.  Perhaps a simple variable option, ie "&test=1" added to the cron command could generate a success message to allow for this.  That, or an option could be placed within ListMail somewhere to put dailymail into and out of 'test' mode.  This would need to be explained in the installation documentation.  I suppose each method is equally useful.  The first method has the added benefit that I don't need to use up any space in the ListMail interface on the Config page.  While I could place the option on the manual Dailymail page it might be confusing for installation.

What do you think about my assumption of 23h55m?

- added queue protections to prevent duplicate emails when refreshing the send page or when resuming queues too soon

This is a really nice improvement that prevents errors (duplicate emails, mostly) and makes ListMail more user-friendly.

When you refresh a standard send page it says "This message has already been sent, did you hit refresh?".  This is achieved by creating a "formid" on each "Send Email" page and storing it in the queue protect table (for 3 days).  I took the caching off of that page so a new formid is always generated.

When you hit Resume on a completed mailing, it says "This mailing has been completed".  This is done by storing the status of a queue which indicates it has completed (3 days).

When a mailing starts, a new entry is made in the new 'queue protect' table with the flag indicating the message is "queueing".  That is, the users haven't been entered into the ListMail queue table yet, which is done before sending to the server.  "Queueing" messages may not be resumed (a message will say "This message is in the process of queueing"), thereby preventing mailing if the ListMail queueing process is interrupted before it finishes.

As any mailing progresses it updates the 'queue protect' table with the current time every 20 seconds.  You may not resume a queue that has responded in less than 60 seconds.

If you try to resume a message that has responded within 60 seconds, a page is shown.  The page tells you how many seconds ago the queue last responded and provides another Resume button so you can try again.  When the last response time is less than or equal to 22 seconds the page states that the queue 'appears to be running normally'.  When the last response time is greater than 22 seconds the page states "seems to be failing".  Once the 1 minute mark is reached you may resume a failed mailing.

I think this improvement will take a LOT of the headaches out of mail sending.  It will also cut down on support requests as the sending process can now be ruled out in case of duplicates, etc.

Do you think I need options for how often the 'queue protect' table is updated and how long after a queue has stopped responding to allow a resume?  Personally, I think 20 seconds and 1 minute is a perfect balance of speed and stability.

- added the ability to move or add followups into the middle of a live sequence, inserting them
 - added 'smart' followup and user seq # renumbering when adding, inserting, and deleting followups

A very useful feature (inserting) and some cosmetic improvements.  After spending an entire night on it I started over, sitting down with a pen and some paper to rewrite the logic.  The result is safer, more straightforward and easy-to-test code.  I have tested this quite thoroughly and am confident there will be no problems with it.

---

That's it for today.  I believe the only other major feature I will implement before the update is an auto resume script.  Here's what I envision:

Scheduled script, ie. "resume.php" runs every 5-15 minutes.  The script checks the 'queue protect' table to see if any mailings have not responded in over 1 minute (or 1.5 - 2 minutes, just to be safer than with manual resumes).  With this feature if a mailing stops responding for any reason (ie. server reboot, MySQL/web/mail service restarts or downtime) your mailing will resume shortly afterwards!

Regards