Exim optimization (CPanel)

[DW]

CPanel is a very common server management system and one I highly recommend for ListMail as it supports it 100%.  It uses Exim email software.

The following has been known to increase outgoing mail speed:

Standard Exim (if your server runs cPanel, scroll down)
By default Exim will only send the first 10 messages you deliver to the server right away.  After that, again by default, it will deliver the rest of your emails an HOUR later.

By making the following change to the Exim config file your messages will be delivered practically as they are queued.

In the file:

Code: [Select]

/etc/exim.conf
Add the following line:

Code: [Select]

smtp_accept_queue_per_connection = 0
If your server has lots of users who are all sending lots of email, you may also want to add this, which enables faster hard drive access for very large outgoing mail queues.

Code: [Select]

split_spool_directory = true
Update: The following lines were suggested by a client further down in this thread.  The configuration options deal with the maximum number of running "sender" processes as well as the number of parallel deliveries per domain.

Code: [Select]

queue_run_max = 10
remote_max_parallel = 15
Don't forget to restart Exim:

Code: [Select]

/etc/rc.d/init.d/exim restart

cPanel with Exim

Login to WHM as root and browse to the Service Configuration -> Exim Configuration page.

Scroll down and click Advanced Editor

Scroll down to the text box beneath #!!# cPanel Exim 4 Config

Add the following lines:

Code: [Select]

smtp_accept_queue_per_connection = 0
split_spool_directory = true
queue_run_max = 10
remote_max_parallel = 15
Scroll way down and click Save

To improve sending performance it may also be a good idea to disable SpamAssassin and other filters on the Exim Configuration page.

[valuehosted]

Dean,

not so sure about this tweak... thought it was odd I didn't get ANY emails for like 5-6 hours... checked my mrtg and it flat lined.  

I had added both of those to my exim.conf file - (restarted it)... had to remove them both and restart and mail has started to flow again.

not sure which one it didn't like though.

--Tone

[DW]

When you restart Exim it begins to process and send any and all email in the queue.  By default, when you connect and start sending messages the first 10 are sent and the rest become "frozen".  The first option I recommend (smtp_accept_queue_per_connection = 0) is, essentially, to ensure that no messages become "frozen" even when you queue a very large number from a single connection.

You should place the option(s) near the top of your /etc/exim.conf file, after the following line:

#!!# cPanel Exim 4 Config

[chufford]

I implemented these server changes as well as the "Primary Key" field in lm_sendq. It made my email blasts go about 10 times faster! An email to 53,000 people used to take 12-16 hours to empty the SMTP queue - today it took about 2 hours!

Thanks for this tip!

[ad]

Hello there,

I cant find any file like /etc/exim.conf on my server.
While there is a folder like "etc", it contains only files like passwd or shadow as ASCII files. No exim.conf

Does this mean I cant change the EXIM configuration on my server?

Thanks

WR

[DW]

This optimization is for system administrators.

As a client on a shared host you are stuck in your home directory and cannot modify the system configuration.

[poppiebrooks]

Ok so if you have cpanel and can't find the file in etc folder, what options do you have to re confiqure the file.  I am a reseller and I can get to the Whm page but it won't allow any view of etc exim files

Thanks
Ed Brooks
http://www.diningpleasure.com
"Bringing Low Cost Dining to Your Desktop."

[DW]

Your only option is to bring the CPanel optimization to the attention of your host.

[idealizm]

I applied this tweak, and my sendouts went from 5-6k hourly, to 15-18k hourly.  GREAT!

Then, after one send-out and a day passed, I did another send-out and noticed I was back at 5k/hr.  I went in SSH via root and saw that the exim.conf file was missing the two line additions that I had made based on your recommendations (above).  

Assuming SOMEONE had changed this file back, I looked at the last few commands entered.  Apparently, my server company admin logged in and did "ps aux", "top", and "kill".  Without knowing too much about unix, I realize that these things list and kill current processes, which shouldn't be anything permanent.  I am assuming that CPanel did an update and the exim.conf file got reset?

REGARDLESS, after I went in, made the exim.conf changes, and restarted exim again... I cannot get the send-out to go blazing fast again.  What could have happened to make this tweak no longer work?  Am I missing something here?  Any suggestions?

I am frustrated that I was able to taste such speed and now I've lost it again.

[DW]

What your host did should indeed have been non-permanent.  It is possible that exim.conf was overwritten by a cPanel update, but I'm not 100% sure - your host could have manually replaced it.  As you suspected, the speed should have improved again after reinserting the modification and restarting Exim.

Unfortunately I do not know why the speed has not improved.  I guess the MySQL server and/or email queue could have been under load.  Has it improved since then?

You might want to re-verify the changes in exim.conf and restart it again with service exim restart or /etc/rc.d/init.d/exim restart

Regards, DW

[DW]

Here's a nice Exim cheatsheet to help with further tweaking!

http://bradthemad.org/tech/notes/exim_cheatsheet.php

[AlanT]

For some reason, my VPS is sending mail at about 500 - 600 PER HOUR!  I envy those who report 5K per hour and more.

I've searched and tried a variety of tweaks that should improve performance, starting with the suggestions here on these forums.  No matter what I try, it seems the only thing that helps a little, is to go into WHM and click the "Attempt to deliver all messages in the queue" about 5 or 6 times.

Is it possible that the type of VPS I'm on would make a difference?  For instance, I'm guaranteed 256MB with a burst limit of 1GB.  I've read that for good mailout performance, you need more memory.  Is that true?

Load averages always seem low, like 0.3 or so.  The 4 CPUs in the system are usually shown as being 97% to 100% idle, even when Exim is processing the queue.  Even the IO wait times seem low.

Are there any other suggestions about how to speed up Exim on a WHM/CPanel server?  I can get into config files directly if I have to.

Thanks.

[DW]

Hi Alan,

You should be able to achieve much more than 5000 per hour.  I assume that you applied the exim.conf changes mentioned in the first post in this thread and restarted exim.

You may want to check that your mail queue is not overloaded.  How many messages are reported when you list the message count?

Code: [Select]

exim -bpc
If this is any more than 10,000 you may want to consider tweaking your retry settings

I wonder if your problems might be caused by Yahoo rate-limiting, which can cause bottlenecks in the queue if the mail server does not recover quickly from a large amount of deferrals.

Check out this post where another client seems to have solved a similar problem, although my response seems to be very similar to what we've already covered here.

Regards

[AlanT]

I will check the Yahoo thing and the other post suggestions.

My exim queue is usually a few hundred messages before a broadcast, with a lot of messages hanging around until they're killed at the 7 day mark.  (At the moment, there are 394 messages in the queue.)

I have less than 4000 subscribers, although many of them are receiving followups, so some of the queued mail is from the followups, although it seems a lot is from leftover spam replies and such.

One day, the server administrators recommend setting default catch-alls to :fail: and the next day they recommend :blackhole:.  I know I was getting a ton of dictionary attacks, but I found an Exim tweak to limit those.  Not sure where I found that one, so if anyone's interested, just Google it.

After posting my request for help, I contacted the server administrators to see if my VPS was being throttled.  Their response was to add two more entries into the Exim config file:

queue_run_max = 10
remote_max_parallel = 15

From what I understand, the remote_max_parallel won't help much in our case as every message is different (subscriber data), but the queue_run_max setting might help.

After they made the change, the delivery rate did seem to pick up a bit.

As suggested elsewhere, I've set the interval between queue runs at 10 minutes instead of the default 60.

I'm also getting a lot of messages saying that I'm using the full 256MB given to my VPS, although I'm allowed to burst up to 1GB if no-one else is using the memory.  The next step up is 512MB, and as expected is almost twice the cost.  I'm not ready to spend $90 a month for hosting just to maintain a 4000 subscriber list.  (Apache is the biggest memory hog. Tried a bunch of tweaks for that too.)

I'll post back when I've had a chance to check on the Yahoo thing.

[DW]

Alan,

I would think a VPS capable of upwards of 50,000 emails per day, but I could be wrong.

It does not sound worth it at all to upgrade the RAM in the VPS.  You might want to try to reduce the memory used, disable the notification emails and/or look for a cheaper fully dedicated server. You should be able to find one with a full gig of RAM for $90/mo.

www.layeredtech.com
www.theplanet.com
www.burst.net
www.iweb8.com

You may also want to consider outsourcing to a shared host.  Host4Profit and ThirdSphere allow a lot of email for far less than $90/mo.
http://listmailpro.com/forum/index.php?topic=1699.0

Regards