Mailing list 100,000 and up

[gwill23]

I have a large mailing list.  The hosting company I am using is mad because I was using a shared server and the bounce.cgi script was crashing the server.  I guess I get too many bounces.  Stinking free email companies.  

Anyway I am working on a solution.  Any help from anyone having similar problems your advise would be appreciated.  

I believe this is why my dailymail is stopping every 5-10k messages or so.

just post a reply to this post with this is me and that way you will be notified any time I update this post.  I'll will let everyone know what is going on.

[gwill23]

I am now up and running on a virtual private server.  Not as expensive as a dedicated server.  However, I may upgrade to a full dedicated server soon.  

I am testing dailymail for a few days and might hav a bounce issue but we will see soon.  I will update this post soon.

A word about migrating from one system to the next.  It can be hard.  Finding out how to dump the mysql database to a .sql file seemed to speed things up for me.  It was an option on my server so I don't know the code.  It would be the same as using a backup feature I think.

[gwill23]

Thank you thank you thank you.  I pegged my server during dailymail.  This caused dailymail to stop processing.  My server statistics are below.  I can almost be certain that anyone sending out more than 20k email on a shared server is pegging the processors.  

No I will admit I think I get a large number of bounces.  While the emails are good.  I have several spammer addresses in my list.  By sending my list out FROM my bounce address all the autoreplys are cleaned off my list.  With a list my size I can't keep up with all the autoreplys and most are just garbage anyway.

First try server
Transfer (MB) 26,000
Burstable Transfer (MBps) 1.5
Disk Space (MB) 4500
Reserved CPU Processes 175
Burstable CPU Speed 2800+  
Reserved Memory (MB) 256

Second try with over 14k email to send
Transfer (MB) 26,000
Burstable Transfer (MBps) 1.5
Disk Space (MB) 4500

Reserved CPU Processes 350
Burstable CPU Speed 2800+
Reserved Memory (MB) 512

I will continue to keep you all updated.

[gwill23]

Dailymail didn't die and I recieved a confirmation email stating the correct number of followups had been sent out.

I still have an issue editing the database.  I hope DW will help me with that soon.

I also, will try sending out to the entire list sometime next week.  This way I can see what happens when I max out my resources.

[BGSWebDesign]

Hi Gunther,

Thanks for keeping us posted here, this is valuable information for anyone running a large list (over 50,000).  I have 120,000 and know that an issue with bounces is going to come up.

If you haven't already seen it, there's one other option, but you might not be willing to use it, as it will slow your send process by hours: use throttle code to throttle the sending procedure.  I've done this with some good luck, what it does is also limit the speed of the bouncebacks.

See this post for more info:
http://listmailpro.com/forum/index.php?topic=32.0

See my Follow Up and Stats post (May 11, 2004), and good luck!
Keep us posted on how it goes.

[gwill23]

35000+ followups sent out last night without any issues.  Dailymail did not stop.  It looks like the issue with dailymail stopping and leaving messages in queue is due to pegging the processes allotted to you.  The throddeling code should work to fix that but I think it might cause dailymail reports to be sent out before the process is finished, and you must be willing to wait for the log process.  My server right now will send out 20000 emails in under an hour.  

DW is looking at a database error I am having as well as possible issues with the bounce.cgi.  We will have to see how it preforms once these two things are fixed for sure.

FYI if you have SSH console access to your system you can execute the TOP C command to watch the processes during a mailing.  This is helpful to see what exactly is happening during the dailymail process.

Gunther Williams
http://www.opportunityone.net

[porl]

You want to push your server then it will bog down and spit the dummy. Slow it down a bit. I am pushing lists of 25,000 at a time and taking 4 hours per list. OK - its slow - but I can leave it and the server will not crash.

[DW]

You should be able to achieve at least 750,000 messages per day but yes, your server can be bogged down.  Performance with very large lists is still being tweaked to prevent this.  I will likely have to rewrite bounce.cgi to be a complete Perl solution, as opposed to passing data to a URL, to offer better performance.  This might mean 2 config files.  config.php and config.pl.  I believe this is where a lot of the problems occur.  I am also considering that there may be MySQL processes that don't die or are increasingly using more memory.

[KB]

I have been watching the posts re large lists. I run upwards of 1 mill now with listmailpro. Lots of trouble with even a dedicated server in the beginning. Require 2 things to fix. Clean supply of lists - that reduces the bounces - and a dual processor.  I was with ev1 - waste of time  - now with a new server supplier and no crashes even if I push the server to way past it's operating limit. Can still only send about 30k ( merging ) an hour. i'd like to hear if anyone is maxing that and how.

Kevin

[mike2]

My List is only around 150K, but I run it normally at 60-80K / hour.  Done with 2 machines, one hosts the DB, the other is a dedicated Mailer running sendmail.

Mike Rogers
Get Extreme Marketing For Your Website:
http://ExtremeMarketingProgram.com

[BGSWebDesign]

Hi Kevin and Mike,

but I run it normally at 60-80K / hour.

Those are amazing numbers, also Kevin the 1Million addresses, I had no idea LMP could handle a list of this size, if either of you could share which host is best for a list of that size I'd like to know (Kevin, you mention that EV1 is not good anymore, which host do you use now?)

Mike, using two boxes and splitting db and sendmail up is great, I never heard of doing that with LMP but sounds great too - surely you're running dedicated?  or leased?  

Kevin, just curious how you do handle bounce processing???  I'm running about 120K bouncing is the next issue I'll have to deal with..

Any info you could provide would be helpful...

[BGSWebDesign]

Hi everyone,

I'm over 120K list size now and need a recommendation on a decent host that can handle this size?  Colocate, shared, virtual private, can someone please tell me which one is best configured for a large list?  

Basically I want to be able to mail to large list sizes 12K-25K and let it go all day, if it has to - it would be best if I could shut down the browser window and have it still run in the background - but I don't even know if that's an option, can someone tell me how they are doing that:

Porl -

25,000 at a time and taking 4 hours per list. OK - its slow - but I can leave it and the server will not crash.

Gunther (gwill23) -

I am now up and running on a virtual private server. Not as expensive as a dedicated server. However, I may upgrade to a full dedicated server soon.

Gunther - do mind me asking which host you are using, and if you're still happy?

Mike2 -

My List is only around 150K, but I run it normally at 60-80K / hour. Done with 2 machines, one hosts the DB, the other is a dedicated Mailer running sendmail.

Which host are you using for something like that, or are you doing it all on your own server - in house?

bauer -

I was with ev1 - waste of time - now with a new server supplier and no crashes even if I push the server to way past it's operating limit.

Kevin, can you please tell me who you would recommend?  I've got to get this server issue handled as I'm spending HOURS baby-sitting LMP and clicking on the 'resume' link!

I can't do that much longer - and DW (Dean) when will see an update/new version with real background processing on sending email?  Or, can you please tell me a way to code that with a PERL or Php task that I can run with CRON?

Getting frustrated 'resuming' all day long!!!!

Sure could use some input...

[mike2]

I am using my own machines on a Dedicated T-1 Line in my own office...

The machines are both relatively low-end Redhat Linux.  The mailer itself, which is probably most important is like a 1.2Ghz P4, with 2 regular 7200RPM IDE drives with a somewhat self-tweaked sendmail.

BTW, I love LMP and Dean has been tremendous, but I've been looking for an update now for almost 1 year with no luck... so lets hope I guess on that update.

[BGSWebDesign]

Hi Mike,

THanks for the follow-up... unfortunately I'm not at the point where I can afford my own T-1 LINE!!!   I need another way to handle the problem and I'm looking at creating a module that will jump start the QUEUE from a PERL module called by CRON.

If anybody else has ideas on this I'd like to know...

[DW]

Most servers do not time out in the middle of sending.  Your host may have a restriction on PHP script loading which would still be reached even with the upcoming background sending option.  A dedicated server should handle over 720,000 messages per day so even a shared host (configured correctly) with 10-20 users should be ok.