Dailymail report is NOT delivered when it crashes...

[BGSWebDesign]

Hi,

I'm running one version back from most current... not sure if you updated this yet but DailyMail report definitely does NOT run if Dailymail crashes - even if I re-start it and it completes - I do NOT get the Dailymail report.

I suggest that you at least WRITE OUT Dailymail report before you begin queueing and sending email, that way we can download the report as .txt file from the appropriate 'reports' folder (if possible).

Otherwise I have a LOT of days with NO Dailymail report and it's very frustrating.

Will this make it into the next version?

[BGSWebDesign]

bump, any word on including this in next release?

[DW]

I'll see what I can do about this one - it will be a bit difficult considering stats could be "lost", etc.  The "numbers", stored in the database or a file, will basically need to be incremented on each and every email, whether starting or resuming, and finally send the report when the queue is complete.

[BGSWebDesign]

Hi DW,

 I'll see what I can do about this one - it will be a bit difficult considering stats could be "lost", etc. The "numbers", stored in the database or a file, will basically need to be incremented on each and every email, whether starting or resuming, and finally send the report when the queue is complete
 

Why?  All you need to do is count the totals up for the mailing and write them to the file BEFORE the queue is filled and the mailing starts.   Are you saying that now you add one to a counter every email that goes out?  (that must be slow).  I don't believe it's necessary to write it to the file every time - just count them before you do the mailing and before you get into the loops?

I know that it may not be accurate based on crashed mailings, etc, but at least we'll have a chance to look at what was supposed to go out.

[DW]

I would prefer to do it as reliably and accurately as possible.  It shouldn't be too intensive to count each and every message.  For speed we might limit a write to every 2 or 3 emails, but it may not be necessary.    MySQL is very good at handling frequent updates, especially with a properly indexed table.  The initial value would only need to be read once at the beginning of a send or resume.

Storing the number of messages sent out of a queue could also speed up the 'items in queue' header.  Currently it counts the number of rows in the queue table, which can be very large.  Doing this often could be intensive.  It would be far easier on the system to query a single entry.

All of this data can likely be stored in the new (1.85) lm_sendp table, which is used to protect queues from early resume and duplicate form submission (ie. refreshing the send page).

I will work on this next

[DW]

I have made progress on this.  The report is created after processing everything, queueing email, etc. but before initiation of the mail send.  This means that if the script crashes when processing bounced messages, for example, the dailymail report will STILL NOT be sent (as might be expected) but it will at least be sent in the case of (more common) SMTP errors or other mailing interruption.

If there are messages to be sent by dailymail the complete report is copied to the queue protect table before initiating the send.  When a dailymail queue is completed, either within dailymail or when resuming, the report is sent immediately following.  If there are no messages to send a queue is not created and the dailymail report is sent by the dailymail script.

Doing it this way we don't have to store how many messages are to be sent and how many are sent or place any additional load on the MySQL server on a per-email basis.

I'm finishing up - this will be in the next update.

Regards

[BGSWebDesign]

Hi DW,

Great that you're working on it, it will help tremendously - though I still wonder why you haven't considered plain text .log and .rpt files instead of sending the Dailymail report?

bounced messages, for example, the dailymail report will STILL NOT be sent

Why?  Have you thought about how often this can happen?  I send out a huge amount of emails/day some days, can you imagine the number of bounces and therefore chances for Bounce Processing to crash?

The other thing I wonder is why don't you consider writing the LMP reports as text files to a folder so we can retrieve them later???  Why is that so difficult?  That would resolve many of these issues.

I appreciate your work and dedication to LMP - but sometimes I feel that you are not deeply in touch with those of us dealing with large list issues - and I can guess at the reason - is it that you have not run your own very large list?   I know that you provide dediated hosting servers for others, so I wonder why you have not come across many of the issues I bring up and others bring up.

The only thing I can think of is because you yourself have not worked with LMP on a constant basis to get an idea of some of the issues we are all dealing with in using the script.    Once you start working with 100-120 lists - 20,000 - 50,000 mailings - and large groups of followup messages (15-20 or more) you will quickly see how some of the things brought up become issues real fast.

I hope that in future you can provide a Poll as others have asked up here many times, going back for years now, that would let US (the users themselves) help decide on issues/importance and what features we want most.   I don't like to berate you on this, but it does seem like it's been asked for by many forum posters, and we never see it?  Why???  We also never see a detailed list of upcoming features/updates something that would tremendously help us figure out how to plan for future mailing possibilities.

Do you think about these things?  If you do, please consider implementing them...

Thanks again for your work, you have a great LMP script - one that could be SO MUCh better if you listened to the input of your many users who use it on a daily basis.

Please don't take this the wrong way, I admire your work and appreicate all that you've done with LMP, I guess I'm just asking for a little more - not that I expect it always, but I feel that you're capable of doing it, you've helped me numerous times - and provided scripts to me quickly that have really helped - I know that LMP could be really, really incredible with some of these extra features: Split Tested Mailings, Send History, Tracked Links tied to purchase tracking, and so forth    

I also feel that you could make LMP provide you with the income you want/need if you implemented some of the things we've talked about up here, 1.) Add-Ons that are purchased, 2.) Subscriber services (I know you're working on that), and 3.) Setup charges for every user requesting assistance...  there is great promise in the future!

[DW]

> bounced messages, for example, the dailymail report will STILL NOT be sent
Why? Have you thought about how often this can happen? I send out a huge amount of emails/day some days, can you imagine the number of bounces and therefore chances for Bounce Processing to crash?

Yes, I would prefer it be sent in this event too, but bounce processing happens before followup processing.  If dailymail is to fail BEFORE followups they will not be processed for that day anyways...  Ideally, and normally, this will _never_ happen.  Note that users using bounce.cgi will have a very low chance of the script crashing, timing out, or getting interrupted before queueing email.  Users not using bounce.cgi (not recommended for lists over 5000 or so) should have relatively few bounces in the mailbox, so should have few problems as far as server timeouts go.  Note that ListMail expects to be run on a host with NO timeouts whatsoever...

I could, I suppose, write the report in increments to try to keep it being sent even when it crashes, but I still don't see a way of re-running dailymail to re-process the SAME DAY in the event of an interruption half-way.  When properly configured the script should not crash or be interrupted.  There are, of course, issues with lengthy processes such as mailings where the MySQL or SMTP service could be 'restarted' or go offline temporarily and this is what my recent fix addresses.

The text file idea is an interesting suggestion - It would not be hard at all to write the report to a file each time a dailymail report is sent.  This is the easy part.  The hard part is with building the report dealing with interruptions.  I believe I have done the best I can considering it is practically impossible to 'recover' from interruptions that occur before (or during) scheduled email queuing and followup processing and queuing.

I know there are difficulties with managing so many lists, especially considering the small sizes and "all-encompassing" nature of the "current list" select boxes.  Along with user selections you might have 200-250 "lists" presented to you such as when sending email.  I wish to somehow overcome the need for select boxes but this may be an unavoidable side-effect of managing such a large number of lists.

Instead of a poll, and as mentioned in another thread, I will organize all suggestions into a list and allow users to vote.  This will come shortly after the new site is introduced, which will likely be completed after the upcoming update.  I will be taking into account all of my personal notes on new features as well as every single suggestion in the development forum.

I value your input.  Please have patience (a lot of what you have requested is coming!) and understand there are some limitations to what I can do such as dealing with unexpected interruptions that occur during, before, or after critical processes.

Regards

[DW]

I've been thinking and figure I _could_ better 'protect' each function of dailymail...  Each dailymail event could be stored when started and the actions logged as they are completed... You might see a note in the header of the program "dailymail didn't finish, click here to resume", or similar.  I'll update this thread with my progress.

Regarding your comments that I could be doing more I was somewhat shocked :shock:.  I understand your frustration with limits but don't see how it's helpful to judge...  I do listen, especially to those who post alot (you:)) and as the days go by I am, quite literally, figuring out how to get what YOU want into the program.

I believe you've agreed with the fact that ListMail should be developed for the masses, and I believe I am doing that with a number of the upcoming improvements, with many more long-awaited features to come.  Implementing the features you want first would be hypocrital, but nevertheless, I can see through your relentless repeated posts that you need and want certain features.  Your technique is a success - I am feeling pressured to satisfy you and, like I said above, I think of ways to best implement your desired features almost every day.  Please be assured that I AM planning on putting at least one of them in very soon!  I would like to apologize to other ListMail clients that I have been pressured - what can I say, I'm a nice guy.

It would be helpful to me if you could simply make your suggestions as straightforward as possible and in one place.  Replying to a new post on another topic with a reminder of something you've suggested 3 or more times becomes, frankly, an annoyance - once or twice is enough!  If you want to bring more positive attention to your suggestion you might open (or continue) a discussion about the benefits.  You could also help with some ideas about ways to work it into the interface or talk about the back-end logic and take some of the load off of your more-devoted-than-ever programmer.  I greatly enjoy discussing new features of any sort but I'd like to keep it organized, in a single thread.  I read _all_ new posts.

Please do not take it personally if it takes some time for certain features.  When I work for free I work for all ListMail owners.  If you would like certain features immediately you might have some success offering payment for custom work.  The saved followup set feature along with scheduling, for example, was a $200 custom feature built out of necessity for a client (a teacher running email courses for students in his/her class).  I do try not to do these jobs too often but I was short on cash and the bills had to be paid.  I enjoy the fact that all ListMail owners can benefit from the addition.

I, unfortunately, do not think I will be able to give you exact time frames for features.  Sometimes things are far more difficult than expected.  Sometimes they're easier.  I believe my time is better spent programming rather than describing the programming...

My spirits are up - I've increased some of my pay-per-click advertising and the response has been great.  I hope it keeps up so that I can remain positive and continue to enjoy taking ListMail to the next level.

Regards

[BGSWebDesign]

Hi DW,

I've been thinking and figure I _could_ better 'protect' each function of dailymail... Each dailymail event could be stored when started and the actions logged as they are completed... You might see a note in the header of the program "dailymail didn't finish, click here to resume", or similar. I'll update this thread with my progress.

Describe in greater detail - this sounds promising!  Any chance it will result in logged action being written to standard .log (text) files?  

Regarding your comments that I could be doing more I was somewhat shocked .

Please don't take it the wrong way, there's really only a few things I wonder about, overall you're doing an excellent job. Also remember, part of the reason for these posting is that I want to see you be SUCCESSFUL at this - I myself went through hard-times for many years, only recently have things been much better - and it feels a lot better too.

I know you can make LMP provide the income you need, if you concentrate on MONETIZING (making money) in the places where it can be made - IMOP 1.) require payment for ANY debugging/install problems - I constantly see you suggesting to 'post  your info' and I'll look at it - that's got to be costing you money, and 2.) Subscriber services for those who want to continue using your services, and finally 3.) 'add-ons' - you could make quite a bit at this part of it...

I love this list of new features and can't wait, especially for these:

- added ability to add, name, save, load, and edit entire sets of followups
- added 'loop followups' list option to automatically restart users at end of sequence to first followup
- added 'reset+load saved followup set when x% users are at end of sequence' list option
- added offsets to date-based message codes, can specify +/- days, months, years, and combinations
- added delay # to user selection searches

Regarding this

Replying to a new post on another topic with a reminder of something you've suggested 3 or more times becomes, frankly, an annoyance - once or twice is enough!

Sorry, I'll keep duplicate posting to a minimum - by now I believe you've heard many times the features I want and feel most important....  that leaves it up to you, if you want further feedback/ideas/someone to bounce ideas off of, just post to the thread - or open a new one on the topic your working on, I'd be glad to comment, provide examples, and even look at sample screenshots and give feedback.  

You could also help with some ideas about ways to work it into the interface or talk about the back-end logic and take some of the load off of your more-devoted-than-ever programmer. I greatly enjoy discussing new features of any sort but I'd like to keep it organized, in a single thread.

I'd be glad to help, you come up with the threads in your 'Development Area', not sure if you have one, or maybe you should set one up, and then I can visit and comment/review and provide feedback on anything you want there?

Please be assured that I AM planning on putting at least one of them in very soon!

Great, can you give me a hint as to which it is - I do appreciate your attention to some of these 'feature requests', and really you will save me an enormous amount of time managing my lists!

The saved followup set feature along with scheduling, for example, was a $200 custom

If you give me an idea on what some of my top 10 jobs would cost me, I'd be happy to consider them, please see the appropriate post where I list them, from what I remember, for example what would you want for 1.) Split Tested Mailings/Subjects,  2.) Better Link Tracking/through to purchases, and 3.) Global Saved sets for the Followups?  Those are most important now, but you'll have to see that post where I list them to be sure... those are the top 3 I can think of now... also realize that some of those are major updates I know that, and I could provide additional beta testing/testing for you as we did with the CRON followup scheduling...

My spirits are up - I've increased some of my pay-per-click advertising and the response has been great. I hope it keeps up so that I can remain positive and continue to enjoy taking ListMail to the next level.

Great Dean, glad to hear it!!  Good for you and I too hope it continues for you - you deserve it!

Let me know what you think about these things we've discussed...

Continued sucess to you and LMP!

[DW]

Brett,

Describe in greater detail - this sounds promising! Any chance it will result in logged action being written to standard .log (text) files?

I assume you were having trouble only during the sending (queuing to server) process, not the preceding LM queuing (insertion to the queue table) operation.   With the recent improvement the log will be emailed to you every single day, whether you have to resume the mailing or not.  Does this solve your immediate problem?  Writing the same report (at the same time) to a text file seems redundant but I could add this for you if you like.

To protect everything down to whether or not followup processing was interrupted mid-queue would be quite a task, especially considering I would like to go as far as to cover every interruption, no matter what.  While I'm at it I would also, of course, like to include such protections in other ListMail operations too.  Due to the large number of often-requested features ListMail is lacking and relative stability (other than with the most time consuming process) of nearly all installations, however, I think further improvements should wait a short time.

Please don't take it the wrong way, there's really only a few things I wonder about, overall you're doing an excellent job.

I've gained a ton of valuable experience over the years but am still learning.  I tend to ignore the praise and feel I can always be better.  Lately I have been working harder than ever on 'everything' which could have been why I was a little sensitive to criticism.  Thanks for taking it lightly.  The biggest problem with me is time management, I suppose.  I work hard, play hard (I'm an avid gamer), I'm working on personal relationships positively and am trying to have something resembling a social life and normal family relations.  ListMail is very important to me and is, in fact, my main source of income.  I was pretty down for a while there when I made some poorly-thought-out choices.  Now I have a great woman (2 years now!) and am working towards a future, a house, and a family with her.   I'm finally happy and there's nothing else for me to do but get motivated and work hard.

Where other efforts have failed ListMail remains tried and true.  I hope/KNOW my recent and continued efforts will pay off.  I'm a little nervous - I've always been very modest/shy around success, so there is a degree of stress-relief involved, but I am managing things more and more positively.  With the goal of perpetual happiness for two (and my bewilderment at my luck!) I'm becoming stronger and more determined. I haven't been this determined in a long time and I apologize for that.  With a little patience (realistically, if I have to set a time frame, by the end of the year) you will see the culmination of all I've learned so far and what I can do with it.

I'm happy to hear that you've been doing well.  I feel that you are, perhaps, one of my most determined clients and I admire that.

1.) require payment for ANY debugging/install problems - I constantly see you suggesting to 'post your info' and I'll look at it - that's got to be costing you money

I'll usually only do this if I feel I can gain some experience from the task and make the program better for everyone based on it.  If something doesn't work as it should for one user it might not work as it should for another.  These free jobs are not very time consuming - I get maybe 2-3 a week, tops and they usually only take a few minutes.  I think a wider range of more appropriately priced support (new site!) and proper marketing will result in more support orders.  When I'm flooded with orders I will have no choice but to express that I do not have time for certain investigations or hire some help to take care of it.

2.) Subscriber services for those who want to continue using your services

Coming, this is practically finished on the new site but I want to get some more features out for you guys first since it's been a few months.

3.) 'add-ons' - you could make quite a bit at this part of it...

Yes this was recently mentioned again in the 'contribution for features' poll thread, I believe, and it got me thinking...  I might offer add-ons, such as versions of the scripts in the customization forum, for sale at a later time, but new features and new site/subscription services/reseller program are first.

4) I plan to integrate some offers right into ListMail.  For example, off-site backup storage.  I was considering a link on the Backup page to "send" backup files to my server(s).  A subscription and generated special 'passkey' entered into config.php or on the Config page would be required and used for authorization.  Furthermore, the process could be automated to backup and send to my server once per day, keeping a maximum size or number of files stored.  All data will be backed up once or even more often to another server in case of server failure.

I really don't like the thought of ads right in the program but the backup service seems like a perfect candidate because it would be so convenient.  If you aren't a subscriber a message could be displayed asking if you would like to subscribe or disable the message.

Future advertising will, for the most part, be restricted to the Help/FAQ pages and the forum, possibly with some contextual/AdSense ads, near the bottom, when viewing topics.

Sorry, I'll keep duplicate posting to a minimum - by now I believe you've heard many times the features I want and feel most important....

Thanks. I apologize for being so blunt - it didn't feel right.  I'm all about respect, especially for my elders which I tend to (wrongfully?) believe most of my clients are, and I'm not very good with criticism - I like best to explore facts creatively.  Then again, I guess these are the types of things that need to be said as a 'manager'...

I'd be glad to help, you come up with the threads in your 'Development Area', not sure if you have one, or maybe you should set one up, and then I can visit and comment/review and provide feedback on anything you want there?

Is there a thread for discussion about link tracking?  Of course there is... I'll have to have a look.  I would like to discuss possible link format choices in the update.  I think I can figure out the rest of it, including the appropriate reports on Sent Messages pages.  There are a couple things to talk about with split testing too, as it is I'm figuring it might be done with User Selection ie "first third of list" "second third of list". We might later 'compare' or 'combine' multiple sent message report pages, perhaps with checkboxes... With a different approach it could be more powerful and less instrusive, however.

Great, can you give me a hint as to which it is - I do appreciate your attention to some of these 'feature requests', and really you will save me an enormous amount of time managing my lists!

I was thinking about doing split testing first.  Link tracking in general needs a major overhaul as it hasn't been touched since listMail's inception. :oops:  I have heard yours and Don's request for per message tracking and agree it would be great for the program.  I've tossed around some ideas on it over the last week or so.  I plan to start revamping link tracking VERY soon, and will likely start working on a better interface and more adequate tracking in the immediate future, if not this update then the next.

Since this threads original topic is (mostly) dealt with I'll soon retire it to 'completed' and create or update others instead.

I should really release what I have so far into General Testing in the next few days so that it can get some exposure.  Some of the changes are to critical processes such as followup processing and renumbering (and user rearranging, which has been a part of v1.85 but the whole process was modified to allow for duplicate seq #s) and they should be tested thoroughly.

Onward, forward.

Regards

[BGSWebDesign]

Hi DW,

Writing the same report (at the same time) to a text file seems redundant but I could add this for you if you like.

I think it's necessary, and will save problems if the report is ever lost... having an archive of the reports as text files would be valuable.

suppose. I work hard, play hard (I'm an avid gamer), I'm working on personal relationships positively and am trying to have something resembling a social life and normal family relations.

Good luck in finding your future... I know that for me gaming never works, it's mostly a 'time waster', and I've got way to many other things to do / think about it.... if you can make it work good for you - but I suggest that you keep a log or something and see how much time you spend gaming - maybe that way you'll see if you think it's worth it, or not?

I'm happy to hear that you've been doing well. I feel that you are, perhaps, one of my most determined clients and I admire that.

Thanks, it has been a hard road, starting back in 1997 when anyone could mail large lists straight from their own desktop, emailing promoting has become very different now - adaptation and change are the key to moving forward....  

but I want to get some more features out for you guys first since it's been a few months.

Great - when is the next release coming?

automated to backup and send to my server once per day

That's a great idea, follow through on it...

Is there a thread for discussion about link tracking?

That's a good question, I was thinking maybe you would setup a 'priority' area on the forum where only Power Users can visit, and post comments, etc, on these type of topics?

there are a couple things to talk about with split testing too, as it is I'm figuring it might be done with User Selection ie "first third of list" "second third of list". We might later 'compare' or 'combine' multiple sent message report pages, perhaps with checkboxes... With a different approach it could be more powerful and less instrusive, however

I don't think allowing User Selection for Split testing has any value - PLEASE just do it by manually splitting the list, for example If I choose Split Test, just ask me - Split list by How Many Users?  100, 1000, or to make it easier on yourself, ask like this 'Split List by [2, 3, 4]' (pulldown box for selecting 2,3 or 4), etc...

Here's WHY it's NOT important for you to think about User Groups for assigning Split Testing:  when a split test mailing is done, it is NOT an advantage to pick/choose users, actually what you want is a RANDOM selection based on 1/2, 1/3, or 1/4th of the list so that you can get a good idea if those users are clicking based on the changes in your split test message - and NOT based on the users that were selected - do you see how it's actually more important to do it randomly?  Do not overthing this and make it too sophisticated, you're going to have a lot of work on the back-end making it work with the Link Tracking (through to purchase) if you really want to get sophisticated - put your time in there...

I was thinking about doing split testing first. Link tracking in general needs a major overhaul as it hasn't been touched since listMail's inception.  I have heard yours and Don's request for per message tracking and agree it would be great for the program. I've tossed around some ideas on it over the last week or so. I plan to start revamping link tracking VERY soon, and will likely start working on a better interface and more adequate tracking in the immediate future, if not this update then the next.

Great - sounds promising - look forward to an incredible ability to track Links through to the actual Purchase would be best - amazing!  Anyway that's probably a major overhaul - please just release what you have, I'll wait for the next version for something that good -

Also, as I mentioned above, consider setting up special Forums area for Power Users to discuss these types of issues?

Thanks again, looking forward to next release...

[BGSWebDesign]

bump...

any answers DW?

[DW]

Brett,

Don't worry I haven't been playing games lately.  I've had enough of their addictiveness over the years.  Now I only play from time to time, such as when I need a quick break from stressful programming.  There's nothing quite like taking out a flamethrower with 32-64 other people running around forming personal vendettas...  It's great stress relief when your brain tires of crunching numbers and logic...  I played DOOM on a 14.4 many years ago and I still play 1st persons today.  Battlefield 1942 (Desert Combat mod) is my fav. (think 1st person with pilotable jets, circling helicopters, parachutes, jeeps and tanks... :shock: Okay, I think my TRUE passion is evident enough... I don't dare buy Battlefield 2... )  Games are pretty much the reason I started learning computers 14 years ago at age 11 (and ever since then my dad's work files haven't been the same. ).  Fear not, the addiction has passed.  (Just say NO to MMORPGS)  I'm in work/life mode now 95% of the time.

to make it easier on yourself, ask like this 'Split List by [2, 3, 4]' (pulldown box for selecting 2,3 or 4), etc...

Yes, I see how this could work.  Then it's just a simple matter of a paginated message editing page.  I think this is the best for computing as splitting by # of users can't be guaranteed to be properly proportioned (easily, nothing's impossible - we just have to warp space/time... )

Here's WHY it's NOT important for you to think about User Groups for assigning Split Testing: when a split test mailing is done, it is NOT an advantage to pick/choose users, actually what you want is a RANDOM selection based on 1/2, 1/3, or 1/4th of the list so that you can get a good idea if those users are clicking based on the changes in your split test message - and NOT based on the users that were selected - do you see how it's actually more important to do it randomly?

I was thinking of selecting the users in order of addition, by auto-incrementing id, then building groups ie, row 1 group 1, row 2 group 2, row 3 group 1, row 3 group 2.  I wonder if selecting this way would, in fact, be more reliable than selecting randomly as it would best spread out the time of subscription across groups?  Hey, we could always use an "Order" option to specify 'Date Added' or 'Random' - are there any others we could do?  Maybe custom fields.

I suppose you don't mind if 2 messages show up under Sent Messages when sending to a group split by 2, or 4 if 4... not 'combining' these will save some coding.  It might also make it easier to quickly check your link statistics (when they are shown).

Consider setting up special Forums area for Power Users to discuss these types of issues?

If we do it forum-based it would basically be as it is now, with numerous threads covering numerous topics... I think it would be best to consider adding a comments section to the upcoming database-driven suggestion voting system on the new site.

Thanks for your patience.

Regards