Author Topic: Postfix optimizations achieve 1.2M emails per day (Read 13206 times)

auto · « **on:** August 01, 2007, 06:42:55 pm »

With the hacks I've done here:

http://listmailpro.com/forum/index.php?topic=1956.0

Plus several tweaks to Postfix to correct the Yahoo issues (postfix actually has a patch for yahoo's recent BS) and also rotating IP's I am able to send out roughly 1.2 million per day with one server.

This is without pissing off my ISP or getting blocked by Yahoo (who accounts for perhaps 20% of my list)

DW · « **Reply #1 on:** August 01, 2007, 09:10:23 pm »

Hi auto,

I am very interested in some details about the Postfix patch. I had assumed that Yahoo's rate limiting was either per-IP or per From/Return-path. Even rotating up to 10 IPs delivery to Yahoo seemed affected beyond tolerable levels. Can you point me toward some information and/or the source of the patch you implemented?

Perhaps Postfix employs more parallel delivery than qmail (which I have most experience with) which could allow it to deliver more email to Yahoo. Were you privy to details about exactly what the problems were and how they were solved? If you can share it could help a lot of ListMail owners.

Regards

auto · « **Reply #2 on:** August 01, 2007, 09:58:59 pm »

Hi Dean,

I don't recall the source of the information. But it fixed the problem.

in /etc/postfix/master.cf add:

Quote

yahoosucks unix - - n - - smtp
-o smtp_connect_timeout=10 -o smtp_helo_timeout=10

in /etc/postfix/transport add:

Quote

yahoo.com yahoosucks:

You might also want to add hotmail or other "deferral" hosts in that file as well.

in /etc/postfix/main.cf add:

Quote

transport_maps = hash:/etc/postfix/transport
deadbeats_destination_concurrency_limit = 50

I also tweaked these settings...your mileage may vary here:

Quote

default_process_limit = 1000
default_destination_recipient_limit = 250
smtpd_error_sleep_time = 0

then of course do a #postmap transport followed by a #postfix reload

That is a quick and dirty "hack"...also you might want to play with the timeouts, ranging from like 5 to 20 instead of 10 to test efficiency.

The Postfix team has also added several new items very recently to directly address these types of problems and improve performance.

You'll really want to upgrade to version 2.4.5, as the last several releases (specifically .3, .4, and .5) have added additional measures to fight the new "deferral" crap that not just yahoo is doing.

Best,
Tara

auto · « **Reply #3 on:** August 01, 2007, 10:17:21 pm »

I also added all this stuff to the bounce.cgi, some of which you had mentioned before and some of which I've found to be some of my own common deferral issues:

Quote

if(index($email,'THIS IS A WARNING MESSAGE ONLY')>-1){ exit; }
if(index(lc $email,'temporarily deferred')>-1){ exit; }
if(index(lc $email,'temporarily off-line')>-1){ exit; }
if(index(lc $email,'temporarily suspended')>-1){ exit; }
if(index(lc $email,'mailbox is full')>-1){ exit; }
if(index(lc $email,'over quota')>-1){ exit; }
if(index(lc $email,'retry timeout')>-1){ exit; }
if(index(lc $email,'timed out')>-1){ exit; }
if(index(lc $email,'try later')>-1){ exit; }
if(index(lc $email,'mailbox temporarily disabled')>-1){ exit; }

Preventing bounces and then allowing postfix to re-send soon after a deferral has corrected the majority of my recent delivery issues

DW · « **Reply #4 on:** August 02, 2007, 05:52:34 am »

Great job - thanks a lot for posting this. I will be taking a close look at all of your changes. I hope to be able to apply the knowledge to other MTAs as well!

Regards :lol:

auto · « **Reply #5 on:** August 02, 2007, 12:12:10 pm »

Glad I could help

Make sure also to switch lm_users to InnoDB and use the transaction hack I posted recently to vastly improve mysql performance if you have a very large lm_users table (I would say over 500k records)

DW · « **Reply #6 on:** August 02, 2007, 06:50:54 pm »

For reference here is that link:
http://listmailpro.com/forum/index.php?topic=1956.0

Do you really think InnoDB helped things significantly? On a site I run with over 5.7M records in a 3.3GB MyISAM database (800MB index), and with insert/update operations happening at several per second, the response time remains pretty good. Of course this is on advanced hardware (4G RAM, 2 HDs + RAID mirroring) While I may not switch LMP to InnoDB right away, one thing is for certain - I need to do some more optimization on the ListMail user database. I have noticed the user database page slowing down with over a few hundred thousand rows, too.

Your success in Yahoo delivery is unique - thanks again for sharing. I hope one day I can figure this all out and be able to apply it to other MTAs.

Regards

auto · « **Reply #7 on:** August 02, 2007, 11:47:07 pm »

I do think that InnoDB made a big difference.

Mainly because it implements row-level locking as opposed to table-level locking.

Therefore it can run concurrent updates...which really speeds things up with the bounce processing, automated scomp removals, and some of my own custom hacks (to segment people onto different lists based on what they clicked on) going full-force.

When my table hit 2 million records it was taking forever to do an update because everything was waiting in line for the lm_users table to unlock...so some queries were taking several minutes.

I'm glad I could help out with the yahoo problem. Let me know if those tweaks fixed your issue.

Best,
Tara