At work on v1.85

[DW]

I've been a little busy with some custom work - one of which is a new feature in ListMail. :-)  A client let me know how to do the "Not Active" query for user selection, but only on MySQL 4.  I tried to work it into User Selection but it became way too confusing to create rules.  What I did was simply added a feature that allows you to enter completely CUSTOM SQL for user selection instead of rules.  How 'bout that?

Greg, I will have to consider how to best implement your feature.  I'm a little swamped right now - I've got to stop jumping on these free modifications all the time and getting sidetracked.  A small donation could get me moving in your direction sooner.

I've got some testing to do as far as optimizing how fast each message is processed and built (a client sending large messages reported slow sending speed) and will then be again working hard on the update.

Brett - another good suggestion and one I have found myself thinking about as I plug away here.  I envision daily, weekly and monthly emails with a start day/date for weekly and monthly mailings.  A monthly mailing with a start date would eliminate the need for many options (drop down to choose the "day" of each month to send) and problems with months having different numbers of days.  I could implement this into the current interface on the Send Email and Scheduler pages.  The options for sending could read "Now", "Date", "Daily" (with days of the week beside it), "Weekly" (with the "start date" beside it), and "Monthly" (also with the "start date" beside it)..  Not too hard, really.  I'll see what I can do.  If I missed something let me know.

I'm really looking forward to getting all these features into the program and making ListMail #1!

Regards

[BGSWebDesign]

Hi DW,

added a feature that allows you to enter completely CUSTOM SQL for user selection instead of rules.

Wow, I love it!  Thanks, just be sure to include plenty of SAMPLES/Examples on how to do the difficult queries...  otherwise this is great!

"Now", "Date", "Daily" (with days of the week beside it), "Weekly" (with the "start date" beside it), and "Monthly" (also with the "start date" beside it).. Not too hard, really. I'll see what I can do.  If I missed something let me know

Ok, here's the trick with this, I'm not sure if your idea will work with how I need to do it, but you can fill me in...   these are NOT weekly, or Monthly, or Daily, or Date, instead they are appearing on a DAY OF WEEK (DOW)

Followup1 - Monday  (regardless of when user joined list)
Followup2 - Wednesday (after the Sunday mailing - followup1)
Followup3 - Sunday (after the Wednesday - of followup2)
Followup4 - Thursday (after the Sunday of followup3)
---- NOTE change of scheduling here ----
Followup5 - Monthly on 3rd of Month
Followup6 - 5 days after Followup5
Followup7 - Monthly on 3rd of Month (hits the next month)
Followup8 - 5 days after Followup7
.... etc, etc, with these, until possibly RESTART With Followup5 (but NOT restarting from the beginning, and NOT including Followup1-4 in the Restart).

Ok, hope that's clear... I know may be interesting to program...

I've got to stop jumping on these free modifications all the time and getting sidetracked.

Yes, you do, they'll eat you up...   Please tell me how much you would need to get started on this Scheduling Update, I need it and it will save me time if you can finish it within a week or so?

Good luck, and keep us posted on how it's going... I'd really like to see a nice 'ETA Post' - with possible idea of when certain features may be ready....  these scheduling features are becoming more important for me....  hope you can help....   man, you're going to have to increase price on 1.85... that's for sure  ; )

[don1]

Dean,

First of all...thanks for your constant improvements.  

I would really enjoy seeing even small updates at a time...it's fun and very useful.

The people that use your program everyday, such as myself, really look forward to any new features and updates...it makes our day. Even one new feature every so often is great...it doesn't have to big a whole new version.... Please upload updates as they come along ...  

I am very happy to say that I made a great investment when I purchased your program.

Thanks for all the hard work!

Don

[DW]

Don, thanks for your suggestion!  I will consider releasing less, more often, in order to keep the hype alive.

Brett,

I can add DOTW, no problem - I forgot that one.

Followup1 - Monday (regardless of when user joined list)
Followup2 - Wednesday (after the Sunday mailing - followup1)
Followup3 - Sunday (after the Wednesday - of followup2)
Followup4 - Thursday (after the Sunday of followup3)
---- NOTE change of scheduling here ----
Followup5 - Monthly on 3rd of Month
Followup6 - 5 days after Followup5
Followup7 - Monthly on 3rd of Month (hits the next month)
Followup8 - 5 days after Followup7
.... etc, etc, with these, until possibly RESTART With Followup5 (but NOT restarting from the beginning, and NOT including Followup1-4 in the Restart).

This could be a bit tricky.

Let's see - we can accomplish followups 1 thru 4 with a simple little addition to signup-xtra.php setting the users' Seq#1 delay to a certain value based on the day of the week.

Followups 5 and 7 are straightforward, a monthly email on a certain day of the month.

As for 6 and 8 - this is the tricky part.    I have an idea: "sub"-scheduled messages - basically a followup attached to a scheduled message... The difficulty / downside of this approach is that we would have to track who received what message and when.  I am not sure I can do this without creating a new table to hold the data.  I wonder how well this type of table data would perform on large (or even medium..) installations where millions upon millions of messages are sent.  I have seen MySQL slow down accessing tables this large without proper server optimizations.  Many servers are not optimized so this may not be feasible for the majority.  Then again, we might be able to store less data somewhere else, such as keeping the user IDs of people due to receive certain messages instead of ones that have already been sent.  While this will still take up space (and a new table) it will perform better under load.  Is it worth creating a whole new table for this one sub-scheduled followup feature? :?:

On a side note: Considering monthly emails further, it IS going to be necessary for extra options to deal with the fact that each month has a different number of days.

ie:
[Send] -> [Monthly]
1st of the month thru 28th (?) of the month
Last day of the month
Last day -1 thru Last day -28
1st Monday thru Sunday
2nd Monday thru Sunday
3rd Monday thru Sunday
4th Monday thru Sunday

That's an overwhelming number of options. What if I simply used a 3 parameter cron-style timing? (omitting hour and minutes or, in the future, just minutes)  That would be a heck of a lot easier to program and work into the interface and would offer far more diverse options that would be exhaustive for me to implement manually (such as every second Sunday in December).  As per my proposal, sub-scheduled messages might also be able to be added to scheduled messages that use cron-style timing.

Regards

[don1]

Don, thanks for your suggestion! I will consider releasing less, more often, in order to keep the hype alive.

Dean,

That's a great way of saying it..."Keep the hype alive".  This kind of stuff does get around on the net, and could help increase sales...that's how I heard about your program to begin with.  Hype on a forum about how good "your personal" service was. and the quality of your program.

Can't wait for the next upgrade..."releasing less, more often"...great idea

Don

[BGSWebDesign]

Hi DW,

What if I simply used a 3 parameter cron-style timing? (omitting hour and minutes or, in the future, just minutes)  That would be a heck of a lot easier to program and work into the interface

I think you're taking this approach way too far...

I can see this being done with simple programming code in the front end of the Admin interface that ADDS the followup to your followup list.  The second scenario (MONTHLY scheduling) is more important to me right now, the DOW scheduling has some value, but a little further down the road...  ok, so maybe implementing the MONTHLY scheduling now and leaving the signup-xtra.php as something to come, or give me quick idea on how to do that, and I'll code it up...

Now to the second scenario (MONTHLY scheduling)

As for 6 and 8 - this is the tricky part.  I have an idea: "sub"-scheduled messages - basically a followup attached to a scheduled message...

I don't see why you would require 'sub' scheduling here?  Basically you just need to look at when the next followup occurs and calculate the number of days until WHEN the next monthly date shows up - why would you need to 'sub schedule'?  and HOW do you see this:

I am not sure I can do this without creating a new table to hold the data

working, HOW would you see this new table working, and what layout would it be?   I have a BSCS degree with concentration in Database Design, so doing things like Normalizing tables for performance are what I'm used to working with... if you tell me what you're tyring to do, I'll spend some time optimizing how you could do that with the tables...  though, I don't believe it's necessary, tell me first why you need to 'sub schedule'?

-------------
UPDATE
------------

What if I simply used a 3 parameter cron-style timing?

Ok, I've thought about your 'CRON' approach a little further, and if that helps you and is EASY to implement then GO FOR IT!   I can work with the cron style format with no problem, and if you can pull it off that way, GREAT!!!!   Basically I guess you're calculating the number of seconds until the next FOLLOWUP based on the CRON type entry - that's fine...  let's see a working example, etc...   as I mentioned, I need this in the next few weeks, if it's possible, let me know, if you need a donation to get moving on this?

[DW]

Brett,

ok, so maybe implementing the MONTHLY scheduling now and leaving the signup-xtra.php as something to come, or give me quick idea on how to do that, and I'll code it up...

The signup-xtra.php would be pretty easy.  You can get a numerical representation of the day of the week with the php date() function.  The users' first delay would be calculated to land on the day you like and the rest of your followups would be set, via standard delay, to go out on certain weekdays afterwards.

Basically you just need to look at when the next followup occurs and calculate the number of days until WHEN the next monthly date shows up

Okay, so the followups table could be modified to contain the id of the scheduled message to send after. The users' 'delay' would be calculated upon reaching that followup based on the followup delay (# days after scheduled message to send) and days until the scheduled message is to be sent. The followup sequence would pause until that message is sent.  I can see this working.

Ok, I've thought about your 'CRON' approach a little further, and if that helps you and is EASY to implement then GO FOR IT!

I think the cron approach would give us some extremely powerful timing options and be reasonably quick to implement.  With it you will be able to do daily, monthly, every 3rd of the month (except february), etc etc. Eventually I could even work some of it into an option-rich interface, using the cron string representation as a back-end, so it's easy for people with little technical knowhow to set up complicated schedules.

As I mentioned, I need this in the next few weeks, if it's possible, let me know, if you need a donation to get moving on this

This is very doable. I can always use a little extra money (still workin' my way out of a small debt but am making progress!) so it would be great if you could pay me for this modification. It will benefit all ListMail owners so I'll do it cheap - $50?  I can start Wednesday and if I work fast I could have it done by Friday/Monday.  Did you want the followup modification too?

Maybe I should release this and the few other features I've implemented before unlimited custom fields..

After that, I should finish and release the new site (set your forum theme to Classic) and work those changes (new help page format, logo, etc) into the next update..

I'm optimistic, motivated, and working hard - I'm sure I'll be out of the red soon

Regards

[BGSWebDesign]

Hi DW,

pay me for this modification. It will benefit all ListMail owners so I'll do it cheap - $50? I can start Wednesday and if I work fast I could have it done by Friday/Monday.  Did you want the followup modification

Sure thing, I've sent the money for update... thanks a million... regarding 'Did I want followup Modification'?   What does that mean, I thought that's what we were doing?

Anyway, I've sent a Contact Email about it to you personally....  Really need the CronStyle Followups, AS LONG AS they can work in succession, from one to the next... etc     I'm sure you get the picture... from the details laid out above...  and in email...

I think the Email I sent was lacking something... really what I need is as follows:
Followup1: First Tuesday of Month
Followup2: 3 days after Followup1
Followup3: First Tuesday of Month (month AFTER Followup1)
Followup4: 5 days after Followup3
etc, etc...

With possibility to CYCLE the followups BACK to First Followup in the list at any time.... but probably at the END of the Followups...  How about a 'RECYCLE Followups' option....

That's it.... email me directly for more, and good luck...

[DW]

I've been working my butt off on this one (Cron Followups).  The interface is done; The error checking is done; The help page is (mostly) done; I have a great cron class all ready to implement and initial testing is showing success.  All I have to do now is put it into dailymail.

I will be putting a few more features in before releasing this to the public as it requires database changes.  I may find a way to release it to Brett first, since he paid me for it.

With possibility to CYCLE the followups BACK to First Followup in the list at any time.... but probably at the END of the Followups... How about a 'RECYCLE Followups' option....

This is possible with a small custom addition noted in this post.  I will make sure to include a "Reset Followups" button to reset all users to the start of the sequence.  I am very likely to put in a loop feature but it might not be right away.

Regards

[BGSWebDesign]

Hi DW,

    Sounds promising... had a look at the Help Page and I'm very pleased...  

One question:

Example: 1,2 3,4 5,6
Result: The 1st or 2nd of March or April, but only if it's a Friday or Saturday

I guess this means I can also do something like:

Code: [Select]

1-7 * 1-5 which would hit the FIRST (and only first) week of any month,  only if it's Monday-Friday...  so, the first Weekday of the month?

Let's say that I want to hit the FIRST Weekday of the month, so I need Monday-Friday, then, I want the NEXT Followup to also hit the NEXT following FIRST Weekday (of the following month)....  how would I do that, would the next followup Cron be the exact same as the Cron above?

One further question... since this is how I'll really need it, the First Weekday of any month to START the sequence, and then a delay of 5-7 days for Followup2, and Then a CRON for the Next First Weekday of the Following Month for the Followup3, and then 5-7 days after for the next Followup4....

Thanks again for your great work, I can't wait to see the update... and yes, I would appreciate getting it first  

[DW]

Code: [Select]

1-7 * 1-5
which would hit the FIRST (and only first) week of any month, only if it's Monday-Friday... so, the first Weekday of the month?

Yes, that's right - although, the weekend can only be 2 days long so you should also be able to use 1-3 * 1-5

Let's say that I want to hit the FIRST Weekday of the month, so I need Monday-Friday, then, I want the NEXT Followup to also hit the NEXT following FIRST Weekday (of the following month).... how would I do that, would the next followup Cron be the exact same as the Cron above?

Yes, that's right, but if the two crons are right next to each other you will need to put in a delay of a few days on the second one to get around the days following the first one.

One further question... since this is how I'll really need it, the First Weekday of any month to START the sequence, and then a delay of 5-7 days for Followup2, and Then a CRON for the Next First Weekday of the Following Month for the Followup3, and then 5-7 days after for the next Followup4....

How is the delay 5-7 days calculated?  Will this just be at your discretion?  There should be no problem doing this.  You can use a cron followup to start it off, a delayed followup second, another cron, and another delayed.

I'm working on getting this feature finished.  Here's my latest issue (I'd like your feedback).

The cron feature matches between given "Last" and "Now" times.  I was thinking that I do not necessarily need to modify the user table to store the "last followup sent" date/time for each user.  Since dailymail is supposed to run every day without fail I could simply run the cron with the "last" time exactly 24 hours before the execution time.  This should work perfectly.  The downside of cron-based followups used in this fashion, however, is that if your dailymail fails to execute on a given day someone could be waiting a while to receive a message they should have received a week or a month before.  Storing the "last followup sent" time for each user will partially prevent this problem because the match will occur next dailymail no matter what.  The remaining problem is if dailymail fails for a number of days in a row and the date skips past matching TWO cron-style followups - again, the user will have to wait until it matches again...  So both methods have a similar drawback but one does not require modifying the user table and has a somewhat higher chance of failure.  What do you think?  Should I avoid modifying the lm_users table? (I'd like to) Is this the ultimate drawback to cron-style followups?  I think this will always be a problem when trying to send on specific days.  The dailymail failure issue might also affect (future) scheduled messages as I don't see how to send "any set for today or BEFORE" with a monthly/weekly mail.

Maybe there's a feature I can create to help smooth things over when dailymail fails.  If I stored when dailymail ran I would be able to tell when it didn't run, and could cycle through previous days "mimicking" the execution on alternate days! Hmmm...  it's like the anti-skip feature for portable CD players. "[X] Automatically "catch-up" when Dailymail fails" on Config page?

I guess I will proceed without modifying lm_users and consider putting in the catch-up feature ASAP.

Regards

[BGSWebDesign]

Hi DW,

Sounds great, ok, here's a few things, first you ask:

How is the delay 5-7 days calculated?

This is an interesting question, initially I thought, well, 5-7 days from actual first send, BUT, if I don't know which day IS the first weekday of th e month, that means I also will NOT know the number of followup days to provide for hitting the NEXT Weekday a week ahead followup, so really what I need is Followup1 - First Weekday of Month, Followup2 - Next available Weekday of Month ONE WEEK LATER (only if that is at least 5 days)... does that make sense, for example, if this month, I hit First Weekday on Tuesday - November 1, 2005, THEN, I want Followup2 to hit next week on Monday November 7, 2005, BUT only IF the number of days between Followup1 and Followup2 is AT LEAST 5 days... so in this example it would be at least 5 days and so ok, so you tell me - would I use CRON and also include DELAY to do something like that - I think I figured it out, see next paragraph...

For another example, if the First Weekday of Month was on a Thursday, for example Thursday December 1, 2005, THEN since the next followup Weekday one week ahead from that is Monday December 5, 2005 IS LESS THAN 5 days out, I need to make an adjustment and wait another day. oh I think I have it, instead I would use a CRON such as this:
"7-13 * 1-5" for the second followup which would guarantee that it occurs in the Second Week but only a weekday : )  Great!!!

... So both methods have a similar drawback but one does not require modifying the user table and has a somewhat higher chance of failure. What do you think? Should I avoid modifying the lm_users table? (I'd like to) Is this the ultimate drawback to cron-style followups?

I think it would be best if you did NOT modify lm_users, adding data there will only INCREASE exponentially the amount of total disk space I use, I have enough subscribers that adding anything there adds quite a bit to my database size....    

I was thinking that I do not necessarily need to modify the user table to store the "last followup sent" date/time for each user. Since dailymail is supposed to run every day without fail I could simply run the cron with the "last" time exactly 24 hours before the execution time.

Yes, I think this is a better option, and I like what you say at the end BEST:

If I stored when dailymail ran I would be able to tell when it didn't run, and could cycle through previous days "mimicking" the execution on alternate days! Hmmm... it's like the anti-skip feature for portable CD players.  "[X] Automatically "catch-up" when Dailymail fails" on Config page?

This is BRILLIANT, good job.... yes, I believe this is the best solution, you STORE when DailyMail last ran - date/time or whatever you need (IN CONFIG or somewhere other than in lm_users, store it only once),  then provide the 'catch-up' option, but tell me HOW would the 'catch-up' option try to 'catch-up'?  You have to be very careful here, does it try to RUN twice instead of ONCE every time it runs, or what do you have in mind for how DailyMail tries to 'catch-up'?

Also, since you're storing when dailymail runs, why not also show on the Config page (Last DailyMail Execution: dayofweek mm/dd/year hh:mm), and if possible, even allow Admin to MODIFY that, that way Admin would have complete control for modifying when next DailyMail runs... or HOW quickly the catch-up tries to 'catch-up'....  is this possible, in other words, if Admin sees DailyMail didn't run for 7 days, being worried about next execution THROWING everything off, I go into Config, turn Catch-Up ON (if it's off), and then set Last Daily Mail Execution 1 MONTH behind... this provides me with a faster 'catch-up', and GUARANTEES that any CRON style Followup will be caught up exactly to the point where I am now  

One requirement of allowing Admin to personally set the Last DailyMail Execution is that they MUST get the correct dayofweek OR it will cause problems...  unless you do NOT let them set that, that would be best, show the last Daily Mail Execution as indicated above, let Admin set the mm/dd/year hh:mm, but do NOT let Admin set dayofweek... then you calculate dayofweek when you do your 'catch-up'...

Ok, that's about it...  sounds great, when will it all be ready/fully tested... I'm happy to wait until Monday next week if required?   Or, if you want me to setup some of my own TESTING here, I'll test it out on my TestList?

Let me know either way...

[DW]

I think I have it, instead I would use a CRON such as this:  "7-13 * 1-5" for the second followup which would guarantee that it occurs in the Second Week but only a weekday : ) Great!!!

Yes, it could work this way or you might be able to do it with a delay of 4 or 5 and a cron task set for any weekday.  I'm tired right now it's hard to imagine.  I should have this ready for you to play with tonight.

HOW would the 'catch-up' option try to 'catch-up'? You have to be very careful here, does it try to RUN twice instead of ONCE every time it runs, or what do you have in mind for how DailyMail tries to 'catch-up'?

I was thinking dailymail would run or loop multiple times faking the 'current date' for all missed days and finally running the current days processing.  This would result in sending multiple followups, in order for all missed days, at next execution of dailymail.  I could extend the 'date' feigning to message codes so date codes are processed based on the previous day(s).  This might be an important option for people with time-based messages.  For example, a "10 day course" where in each message they project the same end date (when you can project dates with the formatted date message code - soon:).

Also, since you're storing when dailymail runs, why not also show on the Config page (Last DailyMail Execution: dayofweek mm/dd/year hh:mm), and if possible, even allow Admin to MODIFY that, that way Admin would have complete control for modifying when next DailyMail runs... or HOW quickly the catch-up tries to 'catch-up'...

Changing this will not effect when the actual server dailymail cron task runs the script.  Apart from being able to modify the 'catch-up' feature (to send the wrong messages) I don't think there is a benefit to allowing modification of the last dailymail time.

if Admin sees DailyMail didn't run for 7 days, being worried about next execution THROWING everything off, I go into Config, turn Catch-Up ON (if it's off), and then set Last Daily Mail Execution 1 MONTH behind... this provides me with a faster 'catch-up', and GUARANTEES that any CRON style Followup will be caught up exactly to the point where I am now

The next execution could easily throw everything off without catch-up on.  I think setting the last execution far before the missed dates would only cause problems.  The catch-up would run with 'incorrect' dates which I envision, as above, also using for message codes.  With catch-up on, running a loop through the days missed will result in the sending of any followups, cron-style or delay-based, just as if dailymail had run on each day.  Cron-style followups should be guaranteed to be caught up correctly.

Ok, that's about it... sounds great, when will it all be ready/fully tested... I'm happy to wait until Monday next week if required? Or, if you want me to setup some of my own TESTING here, I'll test it out on my TestList?

I should be able to give it to you tonight around or shortly after midnight PDT.  All of my testing has resulted in success so far but I still recommend you try the new feature on a test installation.  You can run dailymail manually with various cron-style followup settings to test the results.

Regards!

[BGSWebDesign]

Hi DW,

'date' feigning to message codes so date codes are processed based on the previous day(s). This might be an important option for people with time-based messages.

That's really nice... 'projected message-codes'... I didn't know they were coming, but that's nice too, really nice.... having the 'feigned dates' pop in based on the 'catch-up' would be really, really cool...  great job of programming on that one    :shock:

This is sounding great, just a quick question... I have a further need with this, when you mention:

You want to send an email out not the very next Sunday after the last followup, but the Sunday 2 weeks from then. To do this, set up a Delay of 13 and a cron set for "Any Sunday" which is, as per the example above, * * 0

on your help page....  is it possible to put in a -13 delay to specify that I need this sent out, for example 13 days BEFORE the CRON set runs out... here's an example... FOLLOWUP1 needs to go out 2 weeks BEFORE 1 year elapses, FOLLOWUP2, 3, 4, etc.. then proceed on DELAYS... after FOLLOWUP1.

The real key here, is that FOLLOWUP1 can go out 2 weeks prior to the end of 1 year passing...  is that possible with the way it is setup now?  Can you allow negative values, when CRON is also set?  It would really help, unless you can suggest a better way to do this with a CRON string alone (without me having to write my own code to calculate which month 12 months later would be, then subtract 2 weeks from that date)?  I can't think of any other way to do it, can you?

I should be able to give it to you tonight around or shortly after midnight PDT. All of my testing has resulted in success so far but I still recommend you try the new feature on a test installation

Please take your time... it's not necessary that I have it tonight at midnight, as I don't work on weekends.... so, feel free to play around with this until Monday AM if you like... as far as testing goes, I hope that you've covered all scenarios, as I really want to start it rolling on my current list...  are you suggesting I install an entire Other version of LMP and test on that?   Or that I try to do this by running test CRON Followups on a test list within my current LMP?

[ryang]

Just curious, when do you expect to release the next version? I'm looking forward to some of the new features!