Author Topic: Sending Mail Sometimes Stops Part II  (Read 3241 times)

RowdyRhodes

  • Posts: 48
    • View Profile
    • http://www.fwointl.com/
Sending Mail Sometimes Stops Part II
« on: September 30, 2005, 05:22:11 am »
Hi Dean

I'm still having problems sending mail. (Part I at http://listmailpro.com/forum/index.php?topic=722.0

Actually now I can't sent our newsletter at all without either 1. reducing the size of the email (currently about 30KB) or 2. placing the text online and sending only minimum text in the email and providing links to the HTML pages needed to support the newsletter content. Neither option is going to work for us. After much testing, by both myself and our hosting company, we have come the conclusion that the problem lays with LMP (sort of). The following is from my hosting company tech support people:

"I've seen others (in the LMP forum) questioning about server resources and there's nothing that has already been set or changed that would affect the current performance.  The script appears to be running constantly while it's in sending mode.  This is the problem.  We have a limit on the length of time the script can run on the server.  This is to prevent abuse of resources by any particular individual. Is there anyway you can place a majority of the text in your emails on a page on your website and then include a link to this page in the email?  I really think that if you can cut the size of the email in half, it would greatly increase the speed that these messages are going out.  Thus, reducing the number of times that you would have to resume the process due to the script timing out."

Dean, my response to this to my hosting company is as follows:

"It stands to reason that as the number of people on our list grows the longer the need for the script to stay connected, then at some point the size of the email and the size of the list will meet once again at a time out point, regardless of the size of the email, and I would have to again reduce the size of the email. Is this assumption correct?"

Dean, I'm waiting on an answer, but I know that the answer has to be yes and that even if I go to all the trouble of re-designing our newsletter, I will have the problem again when our list and newsletter reaches a certain size = or > the total resources currently being used to send the newsletter.

Your input would be greatly appreciated. They have monitored the LMP software and when it runs it attempts to use up to 90% of the shared mail server resources and they don't allow that to happen. This is because of the script running constantly. Is there anyway that the script can "chill out" -:) and not run constantly during a send? I thought (as did the hosting company) that's what the reconnect setting was all about. Is the script not disconnecting and then reconnecting? We have tried the setting in there at 1, 100, 500, 1000, and 2000 and it makes no difference on how the software runs and the resources it uses.

Our next mailing is scheduled for Monday. What I'm going to do is break our newsletter in half and turn it into two parts, effectively halving the size of each email and send two instead of one email. If this does not resolve the problem, then I'm stuck without sending capabilities.

I decided to update this with the latest info from our hosting company. I just received the following from them regarding this issue:

"Yes, your assumptions are correct. If your list continues to grow, it would be better to correct the serverside issue rather than make major changes to the content of your emails.

The script is timing out because it's continuously using CPU on the server.  The CPU timeout limit was just recently changed from unlimited to 60 seconds by our lead administrator a couple of weeks ago.  We have been monitoring the effects of this current setting and may increase it to 120 seconds, but there shouldn't be a need to have a script utilizing CPU continuously for a period longer than this.  Generally, most scripts complete within a couple of seconds.  The fact that this script consumes all available CPU when running also affects other users on the same server.

One more thing to ask them is if there are specific PHP settings that would help this script run more efficiently."

Take care,
Rowdy Rhodes
Site Manager
Freelance Writing Organization - Int'l.
- A Writer's Digest Top 101 Site
- #1 Writing Resource Site - The Writer
http://www.fwointl.com/
Publishers of:
Inkwell Newswatch (IN) -- Writer's Ezine
http://www.fwointl.com/in.html and
Writers Site News! Weekly Emailed Newsletter
Freelance Writing Organization - Int'l
http://www.fwointl.com

Free site hosting thousands of writing resources and links. 40+ genres, writers' funding, job listings, education, news, submissions, and more. Publishers of Inkwell Newswatch and Writers Site News. Professional writing resources for writers, authors, editors and publishers.

DW

  • Administrator
  • Posts: 3787
    • View Profile
    • https://legacy.listmailpro.com
Sending Mail Sometimes Stops Part II
« Reply #1 on: October 02, 2005, 07:26:43 am »
Rowdy,

I believe it is normal for running scripts to take up 100% of CPU as they run.  If another process needs the CPU they should be able to use it, too.  Unless they are running more than, say, 20 clients on each server, it shouldn't be too much of a problem.  Of course, some hosts try to run upwards of 50-100 sites on a server, which is a major no-no. ;)

The reconnect setting simply reconnects to SMTP.  It does not currently offer any delay or reduced CPU usage.  A hacked PHP delay _could_ relax processor usage for a short time, but won't effect the elapsed "running time" of the script, which I fear they have (also) limited.

Your host's suggestions to "send less email" or "send smaller messages" are not going to work in the long run.  60-120 seconds is nowhere near enough for ListMail - it takes much longer than that to queue several thousand messages to SMTP.  12 emails per second is a fair estimate.  Sending 100,000 messages (maybe even 10,000) will be impossible on this server.  I highly recommend that you switch to a host (95% of them) that do not have hard limits on CPU usage or scripts.

We might be able to increase the efficiency of ListMail a small amount but the fact is it takes time to queue messages to the SMTP server on any host.  When a client mentions a hard limit on script execution time I always tell them they should switch hosts.

Unfortunately, the host I normally recommend is not good for lists over around 20,000.  If your lists are expected to be smaller than this for some time, sign up as my referral and I'll give you 50% off install. :)  If your lists are larger you might try http://findmyhost.com.  I highly recommend you tell your potential host exactly what you will be using their services for, how many users you will be mailing, etc. so that you have their full support.  You might want to get a new domain so you don't have to move your existing site.

I'm also willing to work with you and your current host...

Regards
Dean Wiebe
ListMailPRO Author & Developer - Help | Support | Hosting

RowdyRhodes

  • Posts: 48
    • View Profile
    • http://www.fwointl.com/
Sending Mail Sometimes Stops Part II
« Reply #2 on: October 02, 2005, 08:51:53 am »
Quote from: "DW"
I'm also willing to work with you and your current host.


Thanks Dean,

I'll pass along a copy of this message to their tech support, see what they say and get back to you. If need be, maybe we should open a ticket on this so that they and you can work together, but let's wait and see what they have to say. I'll get back to this discussion as soon as I hear from them. In the meantime .....

Quote from: "DW"
60-120 seconds is nowhere near enough for ListMail - it takes much longer than that to queue several thousand messages to SMTP. 12 emails per second is a fair estimate.


12/sec = 720/min, so if a list of 10,000 is being sent 1 email it would take approx 14 minutes to queue that list. If the list was 100,000 it would take 140 minutes or almost 2 and a half hours? and when you say queue, do you mean queue or send? Queue in my mind is loading the job and then the mail is sent. Sent is sent. Just want to be clear on time and terminology.

Quote from: "DW"
I highly recommend that you switch to a host (95% of them) that do not have hard limits on CPU usage or scripts.


I'd prefer not to. That's a pile of work to move everything we have to another hosting company (unless you mean just move the mailing software to another host). I've been with my current hosting company 5 years, they have 1.8 million clients, they run a mirrored service in that all their client sites are located in two different cities so that if one building goes down the other kicks in, I've had zero (truly) downtime caused by them since I joined, their pricing is acceptable, and their technical support people amazing (much like you in work ethics and abilities). Maybe they'll reconsider this hard limit CPU usage. This is the first I have heard about that.

Let me know about the above queue clarification while I send off your reply to them. I'll keep you posted (pun intended)  :wink:  :)

Bye for now,
Rowdy Rhodes
Site Manager
Freelance Writing Organization - Int'l.
- A Writer's Digest Top 101 Site
- #1 Writing Resource Site - The Writer
http://www.fwointl.com/
Publishers of:
Inkwell Newswatch (IN) -- Writer's Ezine
http://www.fwointl.com/in.html and
Writers Site News! Weekly Emailed Newsletter
Freelance Writing Organization - Int'l
http://www.fwointl.com

Free site hosting thousands of writing resources and links. 40+ genres, writers' funding, job listings, education, news, submissions, and more. Publishers of Inkwell Newswatch and Writers Site News. Professional writing resources for writers, authors, editors and publishers.

DW

  • Administrator
  • Posts: 3787
    • View Profile
    • https://legacy.listmailpro.com
Sending Mail Sometimes Stops Part II
« Reply #3 on: October 03, 2005, 12:16:37 am »
Quote
12/sec = 720/min, so if a list of 10,000 is being sent 1 email it would take approx 14 minutes to queue that list. If the list was 100,000 it would take 140 minutes or almost 2 and a half hours? and when you say queue, do you mean queue or send? Queue in my mind is loading the job and then the mail is sent. Sent is sent. Just want to be clear on time and terminology.

Yes, it could take this long.  I have known some servers to run just over 800,000 messages per day.

"Queue" means the messages are sent from ListMail to the SMTP server.  If the server is optimized for large amounts of email, final delivery, or at least a first attempt, should occur shortly after queuing.  Properly configured the server should be able to send about as fast as you can queue messages to it.

Quote
Maybe they'll reconsider this hard limit CPU usage. This is the first I have heard about that.

I sure hope they will because I don't think ListMail will run properly on this server otherwise.  I am unsure as to whether or not they are right about the 100% CPU being a bad thing... As far as I know, the CPU, even at 100%, should be able to be used by other applications with just as much priority as the running application.

I could try putting in some PHP delays but that means it will take even longer than estimated to email your list(s).  Plus, if the host has put a hard limit on PHP execution time as well, this will be useless.

I will soon be adding a "queue checker" to automatically resume your mailings that have not responded for, say, 5 minutes.  This could help get your messages delivered on your server but the script will be 'killed' continually by the limitations..

Regards
Dean Wiebe
ListMailPRO Author & Developer - Help | Support | Hosting

RowdyRhodes

  • Posts: 48
    • View Profile
    • http://www.fwointl.com/
Sending Mail Sometimes Stops Part II
« Reply #4 on: October 03, 2005, 09:35:38 am »
Quote from: "DW"
Regards


Thanks Dean. I have forwarded all of our conversation and the information you provided to my hosting company. I am waiting on a reply. Will be back in touch with you soon on this.

Take care,
Rowdy Rhodes
Site Manager
Freelance Writing Organization - Int'l.
- A Writer's Digest Top 101 Site
- #1 Writing Resource Site - The Writer
http://www.fwointl.com/
Publishers of:
Inkwell Newswatch (IN) -- Writer's Ezine
http://www.fwointl.com/in.html and
Writers Site News! Weekly Emailed Newsletter
Freelance Writing Organization - Int'l
http://www.fwointl.com

Free site hosting thousands of writing resources and links. 40+ genres, writers' funding, job listings, education, news, submissions, and more. Publishers of Inkwell Newswatch and Writers Site News. Professional writing resources for writers, authors, editors and publishers.

RowdyRhodes

  • Posts: 48
    • View Profile
    • http://www.fwointl.com/
Sending Mail Sometimes Stops Part II
« Reply #5 on: October 07, 2005, 11:10:13 am »
Hi Dean

More info in on this problem. I'm still scratching my head on this one, as are the programmers at my hosting company. You'll see what I mean below:

---------------------------
Hi Rowdy,

Unfortunately, there's not a lot of good news I can give you.

I might be able to get our lead admin to leave the CPU limit set a few minutes, but we cannot have a script running for an unlimited amount of time.  This limit was actually just put in place about a month ago to help us with another abuse problem we've been having with some other scripts.

Regardless of the new CPU limit, there are still issues with the script itself.  

Prior to the imposed CPU limit, you were still having problems getting it to run completely without stopping.  Even when we've removed the CPU limits to test this script, the script is wanting to use more resources on the server than we consider reasonable.  Most scripts do not require 100% CPU to run.  This is something that even our programmers are having a hard time understanding (about LMP).  

As far as we can tell, this script is just making a couple of queries to the mysql database, generating the email and sending it.  There's nothing that should be using massive amounts of CPU.

Dave

---------------------------

Dean, as you know I have had problems getting LMP to run properly as my list grew over the past year. As of right now we're stuck. Changing to a different host is not a preferred option that's for sure. Sticking where I am, well, that's not good either since I can't properly send mail.

I guess the question revolves around the how and why of LMP actually sending mail through the server. Like they said above "There's nothing that should be using massive amounts of CPU." -- so then why is this happening and what can be done about it?

Is this happening to anyone else out there?

Take care,
Rowdy Rhodes
Site Manager
Freelance Writing Organization - Int'l.
- A Writer's Digest Top 101 Site
- #1 Writing Resource Site - The Writer
http://www.fwointl.com/
Publishers of:
Inkwell Newswatch (IN) -- Writer's Ezine
http://www.fwointl.com/in.html and
Writers Site News! Weekly Emailed Newsletter
Freelance Writing Organization - Int'l
http://www.fwointl.com

Free site hosting thousands of writing resources and links. 40+ genres, writers' funding, job listings, education, news, submissions, and more. Publishers of Inkwell Newswatch and Writers Site News. Professional writing resources for writers, authors, editors and publishers.

DW

  • Administrator
  • Posts: 3787
    • View Profile
    • https://legacy.listmailpro.com
Sending Mail Sometimes Stops Part II
« Reply #6 on: October 08, 2005, 04:49:02 am »
Rowdy,

Quote
I guess the question revolves around the how and why of LMP actually sending mail through the server. Like they said above "There's nothing that should be using massive amounts of CPU." -- so then why is this happening and what can be done about it?

I believe the problem is inherent of mailing lists and is caused by hard-drive access - the writing (queuing to SMTP), reading (sending), and deleting of files (email queue).

I run servers with up to 20 clients and around 1 million total addresses (not emailed everyday, but 200,000 or so are, more on other days).  The CPU usage does go up as email is being queued but the averages stay quite low, percentage-wise.  I believe the averages are a better indicator of when a server is being overworked as opposed to the usage during execution of specific programs.  Of course, your host might know better. :)

I have seen that hard drive speed is the major limiter to server performance.  Access to files seems to slow down greatly when there are folders with thousands files in them, such as is possible when doing large emailings.  To keep the queue size down the host should employ optimizations to the email software to increase the queue folders and rate of sending.  I run qmail with 400 remote connections at a time and can achieve, with very small messages, up to 50 emails/sec.  When my clients send large emails, the server sends them (usually) just as fast as they are queued. Optimizing the mail server this way prevents items from sitting in queue, filling up folders, and causing delayed hard drive access.

I can't help with your host's choice to limit CPU usage but in my (biased;) opinion it is heavy handed.  Instead of limiting CPU, maybe they could hard-limit CGI script (Perl,PHP) maximum execution time to several minutes to prevent scripts from running forever - this would at least let you get out a fair number of emails at a time.  If shell access is granted and a concern/cause of their trouble, I'm not sure what they can do.  Maybe they can limit CPU usage in the shell for that user (chroot/jail environment) Unfortunately, I don't have a lot of experience with these types of limits.

I hope they can figure something out for you. :)  If not, I just found a provider who does dedicated servers (host up to 100 domains with plesk easy to manage and update) for $69/mo with a great affiliate program. :)  Details coming soon.

Regards
Dean Wiebe
ListMailPRO Author & Developer - Help | Support | Hosting

RowdyRhodes

  • Posts: 48
    • View Profile
    • http://www.fwointl.com/
Sending Mail Sometimes Stops Part II
« Reply #7 on: October 08, 2005, 03:10:53 pm »
Quote from: "DW"
Rowdy,


Hi Dean,

Whew! -- and their response is ..............

-----------------------------------------------

There are 2 different issues with the script running on our server.  The first is that it runs continuously while it's gathers and sends the email.  

Second, the observations by us of the amount of CPU it uses while running.

We have not imposed a limit on the percentage of CPU a script can use.  We rarely see a script use this amount of CPU while running, so this leads us to believe that there may be a configuration issue either with the software or operating system that will help to resolve this problem.  The limit that has been imposed is the length of time a constantly running script can use the CPU to accomplish it's task.

The only responses by Dean have been "It works on other servers." and a sales pitch to move to a different host.  The original question was related to hardware/software configuration on the server that will allow this script to run optimally.

We'd love to get this matter resolved so that we can start offering it as an option to other customers.  Either the programmer isn't familiar enough with the server hardware/software to tell us how his program will work best or he's getting a commission to move people to another hosting company.  This is said as an observation and may not be the actual case.  

From what I've read about the software, people have had nothing but good things to say about it and I want to help you make it work on our servers.

I do hope we can get this matter resolved, but more detailed help/info is needed from Dean at LMP.

Dave
-----------------------------------------------

Dean, what's next my friend?
Oh, and Happy Thanksgiving!
I'm spending it with family this year -- hope you have a good one.

Take care,
Rowdy Rhodes
Site Manager
Freelance Writing Organization - Int'l.
- A Writer's Digest Top 101 Site
- #1 Writing Resource Site - The Writer
http://www.fwointl.com/
Publishers of:
Inkwell Newswatch (IN) -- Writer's Ezine
http://www.fwointl.com/in.html and
Writers Site News! Weekly Emailed Newsletter
Freelance Writing Organization - Int'l
http://www.fwointl.com

Free site hosting thousands of writing resources and links. 40+ genres, writers' funding, job listings, education, news, submissions, and more. Publishers of Inkwell Newswatch and Writers Site News. Professional writing resources for writers, authors, editors and publishers.

DW

  • Administrator
  • Posts: 3787
    • View Profile
    • https://legacy.listmailpro.com
Sending Mail Sometimes Stops Part II
« Reply #8 on: October 08, 2005, 09:09:23 pm »
I'd love to be able to be of more help.  I am still of the belief that it is normal operation of the script and the usage is due to sending several emails per second to the server and the server subsequently sending that email, but I could be wrong (and I'm willing to test it).  I have a client running over a million addresses on a server running dual CPUs, which seems to work better for him than a less powerful server he once used.  The truth is, I don't know what the problem is and I cannot guarantee that I can help you get it figured out to your host's liking, but I will try.

I feel a little insulted that your host said I "just" tried to get you to switch hosts and "never" blamed the software, ListMail.  I tried to offer my best advice and feel I offered some useful ideas regarding hardware and software that hadn't previously been thought of.  They should understand that my priority is not that you stay with your host but that you have a successful, working, experience with ListMail, ASAP.  I am here to continue troubleshooting but there are limits, time and skill-wise, to what I can do and sometimes it's even a matter of luck.  I'm not passing blame on to your host (I believe I was very gentle with my point of criticism)...  I understand that they are frustrated and that losing a customer's satisfaction due to what they suspect to be bugs in a script is a touchy subject for them...  I concede that the script is not perfect, but I will always do what I can to improve it.

This issue being questioned opens a door to improving efficiency of ListMail, which everyone can benefit from.  I can write a few test scripts that contain loops, write files, queue emails, etc. to see if the CPU usage is the same, which might give us some clues.  Unfortunately, debugging is not a process that will get your mailing lists working again instantly.  I'll probably be able to get started on this sometime this weekend and could have some useful results next week.

It could take some time, so if you have very important messages to send you might want to get a cheap host, and maybe a new domain, to use temporarily (as I know you mentioned being very happy with your host).  I certainly don't want to put you in the position of choosing between ListMail and this host - I just want you to get your email out. :)

How urgent is your next mailing?  Will this host accomodate you with a higher time-limit (ie 2-3 mins..) so you can get more messages out between resuming?  This could buy us some time.  If your lists and messages are large, however, you will need to resume a number of times and it could become (more) frustrating.  There is also the issue of partially sent messages which could cause a few undelivered emails.

I will do some testing ASAP and hope to have some good news for you soon.  If your host has any tips about tracking CPU usage, such as the methods they use when determining when to terminate the script, I'd be interested in hearing them.

Regards
Dean Wiebe
ListMailPRO Author & Developer - Help | Support | Hosting

RowdyRhodes

  • Posts: 48
    • View Profile
    • http://www.fwointl.com/
Sending Mail Sometimes Stops Part II
« Reply #9 on: October 09, 2005, 08:12:15 am »
Hi Dean,

Quote from: "DW"
I'd love to be able to be of more help.  The truth is, I don't know what the problem is and I cannot guarantee that I can help you get it figured out to your host's liking, but I will try.


I know and I know you're very good at solving problems. I'm not freaking out or anything. Everyone is calm, cool and collected. Just another bump on the cyber-highway my friend.  :lol:

Quote from: "DW"
I feel a little insulted that your host said I "just" tried to get you to switch hosts and "never" blamed the software, ListMail.


It caught me a little off-guard as well. I think two factors caused the remark: one being that they don't know you and they are protecting me, and two, that they have probably heard "it's the host's fault" a million times from software suppliers only to find out that it's the software. Dave (the tech guy there I deal with) I'm sure was not meaning to be insultive, probably just a tad frustrated not being able to resolve this. Personally I'll bet a coffee that it's something simple and a small incompatibility between LMP and their server/operating system. But, we'll see.

Quote from: "DW"
They should understand that my priority is not that you stay with your host but that you have a successful, working, experience with ListMail, ASAP.  


And that! ... in a nutshell ... is why your software is so successful and I want to keep using it. They too want to offer the same level of experience to me and both you and they have always done so. I just happen to be the unfortunate one (for the moment) stuck in the middle!  :)

Quote from: "DW"
This issue being questioned opens a door to improving efficiency of ListMail, which everyone can benefit from.  I can write a few test scripts that contain loops, write files, queue emails, etc. to see if the CPU usage is the same, which might give us some clues.  


Agreed. Whatever is causing this problem will be resolved and once resolved it will benefit everyone here. How you and the host are going to reproduce the problem I don't know unless you both set up a test site on their server. Or is this done using a ticket and you and he discuss it, make tests, then I try and use the changes live? If the latter, please be aware that I only send email once/week and really don't want to innundate my users with test emails to any large degree. I can -- but I will need time (and the ability to send mail) to let them know what is happening.

Quote from: "DW"
Unfortunately, debugging is not a process that will get your mailing lists working again instantly.  It could take some time, so if you have very important messages to send you might want to get a cheap host, and maybe a new domain, to use temporarily.


I understand it'll take time. Our emails are an important source of information and revenue, however there is not much I can do that makes sense. To move, even temporarily to another server, will cause havoc. The send from address would be different, the user's filters would not recognize us and the bouncing would be large. I'll sit tight as long as I know that both you and my host are diligently working on the problem. You two guys are good. That much I know. Teamed together? Well, heck, it'll probably be a week or so and all will be well (keeping my fingers crossed here  :lol: )

I have already asked them to make exception for my mail in an effort to keep my op running. They have always been good to me on these types of things so we'll see. If not, then we'll not send email for a while and bite the bullet in an effort to save LMPkind!  :)  --- for all you other LMP users following this thread, don't forget my sacrifices! hahahahaha

Quote from: "DW"
I certainly don't want to put you in the position of choosing between ListMail and this host - I just want you to get your email out. :)


I can't see it going down that way Dean. The mail will get delivered, one way or another and the show will go on. I have faith in you and Dave. Like a said, a bump in the highway.

Quote from: "DW"
Will this host accomodate you with a higher time-limit (ie 2-3 mins..) so you can get more messages out between resuming?  This could buy us some time.  If your lists and messages are large, however, you will need to resume a number of times and it could become (more) frustrating.  There is also the issue of partially sent messages which could cause a few undelivered emails.


As mentioned, I've asked and we'll see if they will accomodate. More than likely they will and, as also mentioned, I don't mind making some sacrifices to help out and get this resolved for the betterment of both LMP and my host, not to mention being able to get back to what we normally do at our site --- supplying writers with free writing resources. (How's that for working in a little plug? Too conspicuous?  :lol: )

Quote from: "DW"
If your host has any tips about tracking CPU usage, such as the methods they use when determining when to terminate the script, I'd be interested in hearing them.


Your whole message was passed along to them moments ago. I believe Dave is out of the office Sundays and Mondays. So we might hear back by Tuesday. Our mailing is/was supposed to go out tomorrow but it can wait. I'll inform the advertisers and other parties that "Houston, we have a problem" and wait for you and Dave. If you need anything from me, just ask.

And .... before I forget, since you're in BC and I'm in Ont .... Happy Thanksgiving!  :D

Take care,
Rowdy Rhodes
Site Manager
Freelance Writing Organization - Int'l.
- A Writer's Digest Top 101 Site
- #1 Writing Resource Site - The Writer
http://www.fwointl.com/
Publishers of:
Inkwell Newswatch (IN) -- Writer's Ezine
http://www.fwointl.com/in.html and
Writers Site News! Weekly Emailed Newsletter
Freelance Writing Organization - Int'l
http://www.fwointl.com

Free site hosting thousands of writing resources and links. 40+ genres, writers' funding, job listings, education, news, submissions, and more. Publishers of Inkwell Newswatch and Writers Site News. Professional writing resources for writers, authors, editors and publishers.

DW

  • Administrator
  • Posts: 3787
    • View Profile
    • https://legacy.listmailpro.com
Sending Mail Sometimes Stops Part II
« Reply #10 on: October 11, 2005, 04:43:50 am »
Rowdy,

I'm working on it.  I am currently checking into MySQL optimization as I noticed a few errors when using "strace" (a linux 'process' follower/debugger) on my server.  I hope to find some more errors/bugs/efficiency-related-problems and fix them using this tool along with software, documentation and information I can find on the web.  What I learn will later on be useful to server administrators viewing ListMail's 'Optimization' forum.

I have run some test mailings and have not been able to get the CPU load up to a constant 100% even with 3 clients (on a server of 20) sending with ListMail.  Can you ask your host if the CPU usage stays at 100% or if it continues to fluctuate, for example between 50% and 75% and/or 100%?

Allowing someone to login as admin is a potential security breach for ANY administrator, so doing my own hands-on testing on your host is out of the question.  I would like to work with your host on this but I need to know what to tell/ask them.  CPU usage and optimizing servers is not an area I am (yet) a master in, but I am tirelessly plodding along and learning.  Progress is being made surely.

Some of my clients have been reporting script stalling and sluggishness lately, which I should be able to put my finger on with some more testing - maybe it's related. :)

:?: A question: How large are your lists and how large are your messages (copy/paste subject, text, and HTML into a notepad .txt file, save it and check the file size for a rough estimate)

I hope to have some useful information for you and your host very soon.

Regards
Dean Wiebe
ListMailPRO Author & Developer - Help | Support | Hosting

DW

  • Administrator
  • Posts: 3787
    • View Profile
    • https://legacy.listmailpro.com
Sending Mail Sometimes Stops Part II
« Reply #11 on: October 11, 2005, 06:15:45 am »
I have found the following post regarding optimization of MySQL.  It's the most informative and explanatory article I've found on the subject after many hours:

MySQL 4: http://www.databasejournal.com/features/mysql/article.php/3367871
MySQL 3.x: http://www.databasejournal.com/features/mysql/article.php/1402311

Utilizing the suggestions has greatly improved my clients' experience with ListMail on my servers.

I wish I could be sure that the problem is related to 100% CPU usage or the capabilities / efficiency of your server.

So far, I recommend:

1. Making sure that mailings to thousands of users do not sit in the email queue for any length of time, causing slower disk access.  Let me know your server's mailing software and I may be able to help with config optimization.  I am most familiar with Exim (cPanel) and qmail (Plesk) some optimizations are noted in the forum here.

2. The MySQL optimizations noted in the post mentioned above.

When I am sure any/most MySQL inefficiences are resolved (probably quite soon) I will start investigating overall CPU usage.

I would be happy to open dialogue with your host if you would prefer not to act as a go-between - although, I wonder if any other ListMail clients are following this discussion and would prefer we didn't do that:)  Either way, I'll keep everyone updated in this post.

Regards
Dean Wiebe
ListMailPRO Author & Developer - Help | Support | Hosting

DW

  • Administrator
  • Posts: 3787
    • View Profile
    • https://legacy.listmailpro.com
Sending Mail Sometimes Stops Part II
« Reply #12 on: October 11, 2005, 07:07:54 am »
My final optimizations to MySQL:


# Intended for MySQL 3.23.x
# Place this under the heading [mysqld] in the file /etc/my.cnf

# added due to a warning in /var/log/mysqld.log when starting mysqld, default was 1M
set-variable = innodb_additional_mem_pool_size=2M

# larger than *.MYI, calculate with shell command:
# X=0; for f in `du -k /var/lib/mysql/*/*.MYI | awk '{print $1}' | sed s/k//`; do X=`expr $X + $f`; echo $X kb; done;
# i chose 3x the size to allow for expansion
# http://www.databasejournal.com/features/mysql/article.php/3367871
# this was way too low (8MB) and was the major cause of my slowdowns!
set-variable = key_buffer_size=75M

# 1k per 1M ram
# http://www.databasejournal.com/features/mysql/article.php/3367871
# named read_rnd_buffer_size in MySQL 4
# default was way too low
set-variable = record_rnd_buffer=512k

# seems to not matter if it's a bit higher and i needed it
# http://www.databasejournal.com/features/mysql/article.php/10897_1402311_3
# default was 64
set-variable = table_cache=128

# default was 50
# http://www.databasejournal.com/features/mysql/article.php/10897_1402311_3
set-variable = back_log=300

# default was 2MB!
# http://www.databasejournal.com/features/mysql/article.php/10897_1402311_4
set-variable = sort_buffer=16M



Don't forget to restart MySQL... :)
Dean Wiebe
ListMailPRO Author & Developer - Help | Support | Hosting

DW

  • Administrator
  • Posts: 3787
    • View Profile
    • https://legacy.listmailpro.com
Sending Mail Sometimes Stops Part II
« Reply #13 on: October 11, 2005, 07:24:48 am »
We're taking this over to email... :)
Dean Wiebe
ListMailPRO Author & Developer - Help | Support | Hosting