ListMailPRO Email Marketing Software Forums
ListMailPRO Email Marketing Software Forums => Server Optimization, Tweaks => Topic started by: gwill23 on August 20, 2004, 10:00:12 am
-
I have a large mailing list. The hosting company I am using is mad because I was using a shared server and the bounce.cgi script was crashing the server. I guess I get too many bounces. Stinking free email companies.
Anyway I am working on a solution. Any help from anyone having similar problems your advise would be appreciated.
I believe this is why my dailymail is stopping every 5-10k messages or so.
just post a reply to this post with this is me and that way you will be notified any time I update this post. I'll will let everyone know what is going on.
-
I am now up and running on a virtual private server. Not as expensive as a dedicated server. However, I may upgrade to a full dedicated server soon.
I am testing dailymail for a few days and might hav a bounce issue but we will see soon. I will update this post soon.
A word about migrating from one system to the next. It can be hard. Finding out how to dump the mysql database to a .sql file seemed to speed things up for me. It was an option on my server so I don't know the code. It would be the same as using a backup feature I think.
-
Thank you thank you thank you. I pegged my server during dailymail. This caused dailymail to stop processing. My server statistics are below. I can almost be certain that anyone sending out more than 20k email on a shared server is pegging the processors.
No I will admit I think I get a large number of bounces. While the emails are good. I have several spammer addresses in my list. By sending my list out FROM my bounce address all the autoreplys are cleaned off my list. With a list my size I can't keep up with all the autoreplys and most are just garbage anyway.
First try server
Transfer (MB) 26,000
Burstable Transfer (MBps) 1.5
Disk Space (MB) 4500
Reserved CPU Processes 175
Burstable CPU Speed 2800+
Reserved Memory (MB) 256
Second try with over 14k email to send
Transfer (MB) 26,000
Burstable Transfer (MBps) 1.5
Disk Space (MB) 4500
Reserved CPU Processes 350
Burstable CPU Speed 2800+
Reserved Memory (MB) 512
I will continue to keep you all updated.
-
Dailymail didn't die and I recieved a confirmation email stating the correct number of followups had been sent out.
I still have an issue editing the database. I hope DW will help me with that soon.
I also, will try sending out to the entire list sometime next week. This way I can see what happens when I max out my resources.
-
Hi Gunther,
Thanks for keeping us posted here, this is valuable information for anyone running a large list (over 50,000). I have 120,000 and know that an issue with bounces is going to come up.
If you haven't already seen it, there's one other option, but you might not be willing to use it, as it will slow your send process by hours: use throttle code to throttle the sending procedure. I've done this with some good luck, what it does is also limit the speed of the bouncebacks.
See this post for more info:
http://listmailpro.com/forum/index.php?topic=32.0
See my Follow Up and Stats post (May 11, 2004), and good luck!
Keep us posted on how it goes.
-
35000+ followups sent out last night without any issues. Dailymail did not stop. It looks like the issue with dailymail stopping and leaving messages in queue is due to pegging the processes allotted to you. The throddeling code should work to fix that but I think it might cause dailymail reports to be sent out before the process is finished, and you must be willing to wait for the log process. My server right now will send out 20000 emails in under an hour.
DW is looking at a database error I am having as well as possible issues with the bounce.cgi. We will have to see how it preforms once these two things are fixed for sure.
FYI if you have SSH console access to your system you can execute the TOP C command to watch the processes during a mailing. This is helpful to see what exactly is happening during the dailymail process.
Gunther Williams
http://www.opportunityone.net
-
You want to push your server then it will bog down and spit the dummy. Slow it down a bit. I am pushing lists of 25,000 at a time and taking 4 hours per list. OK - its slow - but I can leave it and the server will not crash.
-
You should be able to achieve at least 750,000 messages per day but yes, your server can be bogged down. Performance with very large lists is still being tweaked to prevent this. I will likely have to rewrite bounce.cgi to be a complete Perl solution, as opposed to passing data to a URL, to offer better performance. This might mean 2 config files. config.php and config.pl. I believe this is where a lot of the problems occur. I am also considering that there may be MySQL processes that don't die or are increasingly using more memory.
-
I have been watching the posts re large lists. I run upwards of 1 mill now with listmailpro. Lots of trouble with even a dedicated server in the beginning. Require 2 things to fix. Clean supply of lists - that reduces the bounces - and a dual processor. I was with ev1 - waste of time - now with a new server supplier and no crashes even if I push the server to way past it's operating limit. Can still only send about 30k ( merging ) an hour. i'd like to hear if anyone is maxing that and how.
Kevin
-
My List is only around 150K, but I run it normally at 60-80K / hour. Done with 2 machines, one hosts the DB, the other is a dedicated Mailer running sendmail.
Mike Rogers
Get Extreme Marketing For Your Website:
http://ExtremeMarketingProgram.com
-
Hi Kevin and Mike,
but I run it normally at 60-80K / hour.
Those are amazing numbers, also Kevin the 1Million addresses, I had no idea LMP could handle a list of this size, if either of you could share which host is best for a list of that size I'd like to know (Kevin, you mention that EV1 is not good anymore, which host do you use now?)
Mike, using two boxes and splitting db and sendmail up is great, I never heard of doing that with LMP but sounds great too - surely you're running dedicated? or leased?
Kevin, just curious how you do handle bounce processing??? I'm running about 120K bouncing is the next issue I'll have to deal with..
Any info you could provide would be helpful...
-
Hi everyone,
I'm over 120K list size now and need a recommendation on a decent host that can handle this size? Colocate, shared, virtual private, can someone please tell me which one is best configured for a large list?
Basically I want to be able to mail to large list sizes 12K-25K and let it go all day, if it has to - it would be best if I could shut down the browser window and have it still run in the background - but I don't even know if that's an option, can someone tell me how they are doing that:
Porl - 25,000 at a time and taking 4 hours per list. OK - its slow - but I can leave it and the server will not crash.
Gunther (gwill23) - I am now up and running on a virtual private server. Not as expensive as a dedicated server. However, I may upgrade to a full dedicated server soon.
Gunther - do mind me asking which host you are using, and if you're still happy?
Mike2 - My List is only around 150K, but I run it normally at 60-80K / hour. Done with 2 machines, one hosts the DB, the other is a dedicated Mailer running sendmail.
Which host are you using for something like that, or are you doing it all on your own server - in house?
bauer - I was with ev1 - waste of time - now with a new server supplier and no crashes even if I push the server to way past it's operating limit.
Kevin, can you please tell me who you would recommend? I've got to get this server issue handled as I'm spending HOURS baby-sitting LMP and clicking on the 'resume' link!
I can't do that much longer - and DW (Dean) when will see an update/new version with real background processing on sending email? Or, can you please tell me a way to code that with a PERL or Php task that I can run with CRON?
Getting frustrated 'resuming' all day long!!!!
Sure could use some input...
-
I am using my own machines on a Dedicated T-1 Line in my own office... :D
The machines are both relatively low-end Redhat Linux. The mailer itself, which is probably most important is like a 1.2Ghz P4, with 2 regular 7200RPM IDE drives with a somewhat self-tweaked sendmail.
BTW, I love LMP and Dean has been tremendous, but I've been looking for an update now for almost 1 year with no luck... so lets hope I guess on that update.
-
Hi Mike,
THanks for the follow-up... unfortunately I'm not at the point where I can afford my own T-1 LINE!!! I need another way to handle the problem and I'm looking at creating a module that will jump start the QUEUE from a PERL module called by CRON.
If anybody else has ideas on this I'd like to know...
-
Most servers do not time out in the middle of sending. Your host may have a restriction on PHP script loading which would still be reached even with the upcoming background sending option. A dedicated server should handle over 720,000 messages per day so even a shared host (configured correctly) with 10-20 users should be ok.
-
Dean,
Most servers do not time out in the middle of sending.
Your host may have a restriction on PHP script loading which would still be reached even with the upcoming background sending option.
I don't believe this is the case, as sometimes it can complete all the way through the mailing without showing the RESUME option, how could that be if the problem were related to PHP script loading?
Also, I use SMTP server mailing (reconnect every 2500 emails), not PHP, is that part of the problem, should I set RECONNECT every 2500 emails to a lower number, say 500?
-
I was loosing messages a while back. I found that my server only allowed 1000 messages per connection. I changed the "reconnect" setting to 999 and that made all the messages work.
There are some other interesting optimizations discussed at
http://listmailpro.com/forum/index.php?topic=49.0
-
Hi Guys
I've just seen some of your queries about where I host. I've mailed my own server guys to see if they want more mailing business.They will respond shortly - if they are happy I'll post their details.
Regards
Kevin
-
Hi Kevin,
Thanks, I'd like to know if they are available, to have an option for emailing larger lists with a host that knows how LMP works :P
Keep us posted please...
-
Hi guys.
I have been in contact with the guy who hosts my listmail stuff. He provides a dedicated server that never falls over and gives me between 500 and 600k a day. Also helped me move my stuff from ev1
My server runs $300 a month
He's happy to talk with you - phoned me in South Africa to answer some of my questions. Has reverse dns so so you can whitelist at aol. If you want his details then mail me at bauer@mweb.co.za with your details and I'll pass them on.
Regards
Kevin
ps. He does have smaller systems so maybe they would be suitable for smaller lists
-
I don't have a list anywhere close to those numbers, but I can't even send out to 5000 a day on H4P.
What I do have is my own server for website hosting, so I'll be moving from H4P ASAP. What is the best way to move my subscribers?
I have a lot of space and bandwidth to offer for a GREAT fee, for LMP owners only. If you will send me a PM and tell me what you are looking for, I will reply with my Special LMP Price.
Neil
-
Dean,
Is the throttling code still recommended int he newest version of LM to assist in the sending of lists larger than 10K?
Please provide as much detail as possible regarding when and if the code should be used.
Thanks,
Dane
Most servers do not time out in the middle of sending. Your host may have a restriction on PHP script loading which would still be reached even with the upcoming background sending option. A dedicated server should handle over 720,000 messages per day so even a shared host (configured correctly) with 10-20 users should be ok.
-
Dane,
I don't recall why this was done but I do not recommend throttling code be implemented with SMTP. I have not found it to be necessary at all. If your list is over 25,000 or so I recommend that you ask your host to make sure the server is optimized so your messages do not slow down the server. (I would also reassure them that you run a very professional list). You can already delay "internal PHP mail" if you use that for your list email.
-
Neil,
The best way to move your subscribers - in fact, your entire ListMail database, is through PhpMyAdmin. Do an "Export" of all of your tables with the "Complete Inserts" option enabled. Save it to your local computer and re-import it into a new database on the new host via PhpMyAdmin or the command-line.
Regards
-
Dean,
Are there any special set-up features to run larger lists, 10K - 250K in size?
Thanks,
Dane
You should be able to achieve at least 750,000 messages per day but yes, your server can be bogged down. Performance with very large lists is still being tweaked to prevent this. I will likely have to rewrite bounce.cgi to be a complete Perl solution, as opposed to passing data to a URL, to offer better performance. This might mean 2 config files. config.php and config.pl. I believe this is where a lot of the problems occur. I am also considering that there may be MySQL processes that don't die or are increasingly using more memory.
-
I have experience with qMail (Plesk) and Exim (cPanel, Ensim) and have posted a few optimizations for these mailers. Your server administrator could make use of what is mentioned in these posts. You may want to forward the information to them.
There is one further optimization that may not be in effect on your installation: Try adding an index on the id field of the lm_sendq table with PhpMyAdmin.
Regards
-
Great topic.
Good input from all.
Our primary list topped 405K this week. Big. Very big.
I also just moved to a new server w/Cpanel- so great timing DW with
the LMP import fix.
It seems like everyone's working pretty hard to manage the bigger lists, between throttling, primary key fixes (maybe from another post), running on two machines...
We're not very technical, so we needed a clean, simple solution. Knowing that we've run successfulyl with lists in the 30-40K range, I decided to take our master list and split it by 50K each, done alphabetically with Mailloop's listp processor, and catalogued alphabetically to keep track of where we were within the database.
So far so good. Looking at it now we sitting at 2.11 emails per second on the first list. I plan to queue them up seperately and run each list individually.
Just wanted to share what we've done, thought I would sure be interested in any easy to implement solutions to tweak, modify or seriously expedite things to reach the 10-12K per-hour mark.
Thanks for all your input.
Andrew
-
There is one further optimization that may not be in effect on your installation: Try adding an index on the id field of the lm_sendq table with PhpMyAdmin.
My apologies for resurrecting an old thread, but I need clarification.
I am using ListMailPro version 1.87. Looking at the lm_senq table, it seems like there is already a primary key for the id column set up. So, do I take it to be optimized already?
-
Stefaans,
Yes, that's right. :)
Regards