Author Topic: Delayed sending and Deferred?  (Read 28949 times)

BGSWebDesign

  • Posts: 625
    • View Profile
    • http://www.bgswebdesign.com
Delayed sending and Deferred?
« on: March 19, 2015, 06:34:13 pm »
Hi,

Referring to this Exim config post:
http://listmailpro.com/forum/server-optimization-tweaks/exim-optimization/

I'm running on a VPS server with Exim, as follows:     
--
from rootuser by server.mydomain.com with local (Exim 4.82)
--

Now, here's the problem.  I've run with this same configuration, and this same server for the last 3 years.

Everything was fine, until about 10-15 days ago.  Now, what is happening is that I'm ending up with thousands of
emails jammed int he Mail Queue from the RootUser account at the mailserver at my domain, and then I end up getting
these emails saying 'Delayed Sending', or 'Delay in Sending', or something to that effect as reply emails back to my same
rootuser account, AND to my bounce email account - which I am using to remove bounces from every day (through Dailymail).

I just now tried to send the whole jammed up queue (about 28,000 messages), not a huge batch, but I see when sending I'm getting
ALL of these replies talking about 'deferred due to user complaints' coming from Yahoo.  Such as this:
--
SMTP<< 421 4.7.0 [TS01] Messages from xx.xx.xx.xxx temporarily deferred due to user complaints - 4.16.55.1; see http://postmaster.yahoo.com/421-ts01.html
LOG: MAIN
--

When you visit that Postmaster page you see this:
https://help.yahoo.com/kb/postmaster/SLN3434.html?impressions=true

Which seems to indicate it could be temporary, or maybe not, and lists all kinds of remedies, such as DKIM, and several others.

Now, what is strange is that I've been running this server for quite a long time. Over 10 years, so for them to start reporting this repeatedly is frustrating as hell.

So, my question to you DW, or anyone else sending larger volumes of email.

What is the best way to solve this problem with Yahoo???   I'm using SMTP sending directly from my own VPS server. I have reverse DNS setup, but DKIM may be missing
from some messages. 

So, is that the whole problem - DKIM? 

What to do about this???  Is there a solution that already has the servers optimized for sending, and can handle these kinds of goofy complaints without me having to deal with them every time they come up?

Thanks,
-Brett
http://www.bgswebdesign.com/Contact-Us.php

*** I do custom List Mail Pro installations ***
Contact me through my website (above)

DW

  • Administrator
  • Posts: 3787
    • View Profile
    • https://legacy.listmailpro.com
Re: Delayed sending and Deferred?
« Reply #1 on: March 19, 2015, 07:57:51 pm »
Hi Brett,

I think your best option is to contact Yahoo through their recommended channels and apply for whitelisting.

Did you see/remember this post?

Here's a DKIM hack you might employ if your server doesn't support it system-wide.

Regards
« Last Edit: March 19, 2015, 08:01:01 pm by DW »
Dean Wiebe
ListMailPRO Author & Developer - Help | Support | Hosting

BGSWebDesign

  • Posts: 625
    • View Profile
    • http://www.bgswebdesign.com
Re: Delayed sending and Deferred?
« Reply #2 on: March 19, 2015, 08:05:30 pm »
Hi DW,

Thanks - appreciate your fast reply! 

It appears it's related to DKIM.  I know I set someone else up on this several years ago, so looks like I'll have to dig back into it.  Do you have a quick/easy fix on how to setup DKIM properly on EXIM server for LMP?

Is it ok to include that LMP DKIM fix even if DKIM is supported on my server? I do see the configuration places for DKIM in my WHM management panel (for EXIM).  If I set that up in there, do I also setup the LMP DKIM fix you suggest?

« Last Edit: March 19, 2015, 08:15:03 pm by BGSWebDesign »
Thanks,
-Brett
http://www.bgswebdesign.com/Contact-Us.php

*** I do custom List Mail Pro installations ***
Contact me through my website (above)

DW

  • Administrator
  • Posts: 3787
    • View Profile
    • https://legacy.listmailpro.com
Re: Delayed sending and Deferred?
« Reply #3 on: March 19, 2015, 08:22:55 pm »
Hi Brett,

If DKIM is configured server-wide you wouldn't use the LMP hack. I'd recommend Google if you have any questions about doing that with WHM, etc.

Regards
Dean Wiebe
ListMailPRO Author & Developer - Help | Support | Hosting

BGSWebDesign

  • Posts: 625
    • View Profile
    • http://www.bgswebdesign.com
Re: Delayed sending and Deferred?
« Reply #4 on: March 22, 2015, 09:52:33 am »
Hi DW,

Quote
If DKIM is configured server-wide you wouldn't use the LMP hack. I'd recommend Google if you have any questions about doing that with WHM, etc.

I see, yes, it's much easier enabling DKIM through cPanel then trying to do all of this in LMP as I did previously.  Much, much easier, you just flip a switch in Email Authentication. 
What I found in there was that I had some old Mailjet DKIM keys setup that were causing my DKIM to fail.  I removed those and all appears fine now.

However, it appears that my system email account (my root username), is receiving every delay and  standard sendmail bounce that is sent. (in was up to 3.5 GB when I had
it cleared, but now it's back up to 9.5 MB in just a few days. It is full of 'delayed response' messages from the Exim mail queue, as well as bounced messages from the standard
sendmail. This is a standard root username email address, not one that I use at all in LMP Configuration.

What is the best method to not have that email collected at the system level, and instead just have it ignored? 

It seems that I have the option for 'default address' (in cPanel) set to just delete emails coming for catch all - so it's not that, but it appears this root level system account is used
to receive any kind of Exim mail queue bounces, or sendmail bounces, etc.

How do I stop that email address from filling up - what is the best way, that will not affect LMP?
(I could setup a CRON task to just delete all the mail in that folder every night, but I thought you might know of a better way to handle this?)

« Last Edit: March 22, 2015, 09:54:29 am by BGSWebDesign »
Thanks,
-Brett
http://www.bgswebdesign.com/Contact-Us.php

*** I do custom List Mail Pro installations ***
Contact me through my website (above)

DW

  • Administrator
  • Posts: 3787
    • View Profile
    • https://legacy.listmailpro.com
Re: Delayed sending and Deferred?
« Reply #5 on: March 22, 2015, 01:03:14 pm »
Hi Brett,

I know sendmail sends annoying messages warning about a delay when a message can't be delivered, but I haven't seen it on Exim.

Something like:

Quote
Warning: could not send message for past 4 hours

Per this page there are a couple configuration options that might be changed.

I might try setting the TO_QUEUEWARN higher than the TO_QUEUERETURN to skip the warnings altogether. I don't know if this works or not. e.g.

Code: [Select]
define(`confTO_QUEUEWARN', `8d')dnl
define(`confTO_QUEUERETURN', `7d')dnl

You would set that up in sendmail.mc then run "sendmailconfig" and hit enter a bunch of times or compile it manually like this.

Hope it helps.
Dean Wiebe
ListMailPRO Author & Developer - Help | Support | Hosting

BGSWebDesign

  • Posts: 625
    • View Profile
    • http://www.bgswebdesign.com
Re: Delayed sending and Deferred?
« Reply #6 on: March 23, 2015, 07:44:55 pm »
Hi DW,

Thanks, that's just what I needed...  I'm going to decide if it's worth it to re-configure sendmail, but I may instead just setup a CRON script to delete all those delay sending emails, as they don't appear to be worth keeping.

Everything else seems to be back to normal, the odd thing is that this all started only 3-4 weeks ago, before that my system was running fine for the last 3 years on the same server...  I suspect some change appeared for configuration of either sendmail, or exim in the last few months that is contributing to the huge amount of 'delayed sending' emails... 

Such is a day in the life of a mail list server manager  :P  It's exhausting sometimes!

Do you have any ETA on when the next/new version of LMP may be coming out - I  know you've been speaking of working in the code recently?

Thanks,
-Brett
http://www.bgswebdesign.com/Contact-Us.php

*** I do custom List Mail Pro installations ***
Contact me through my website (above)

BGSWebDesign

  • Posts: 625
    • View Profile
    • http://www.bgswebdesign.com
Re: Delayed sending and Deferred?
« Reply #7 on: April 26, 2015, 03:23:15 pm »
Hi DW,

I have enabled DKIM again, and verified it is working, but deferred Yahoo messages are still pouring in, by the thousands. 

I'm not sure how to solve this, other than to desperately try to get something through to Yahoo Postmaster to verify/validate my SMTP sending IP address, does this sound right?  It appears Yahoo has made many changes in how they handle Feedback Loops and other things, see this (on avoiding deprioritization):
https://help.yahoo.com/kb/guidelines-avoiding-deprioritization-sln3443.html

This on PostMaster Basics (and FBLs):
https://io.help.yahoo.com/contact/index?page=contact&locale=en_US&y=PROD_MAIL_ML#

This for recommended Guidelines for Bulk Mail Sending:
https://help.yahoo.com/kb/recommended-guidelines-bulk-mail-senders-postmasters-sln3435.html

Where it clearly says these 421 errors are temporary:
Quote
Retry 4xx messages - This is a temporary error.

And to make sure that FBLs are in place:
Quote
Enroll in our feedback loop - If you're signing your emails with DKIM or DomainKeys, our Complaint Feedback Loop program can help you track and manage your spam complaint rates.

Yahoo Mail Complaint Feedback Loops:
https://help.yahoo.com/kb/complaint-feedback-loop-program-sln3438.html

Finally, FBL signup program:
http://feedbackloop.yahoo.net/

Where these steps are indicated:
--
Yahoo! offers a Complaint Feedback Loop service (now handled through Return Path), free of charge, via this site operated by Return Path. To begin the process:

1.) Determine your sending domain (the d= value in your DomainKeys or DKIM signature).
2.) Determine your selector (the s= value in your DomainKeys or DKIM signature).
3.) Create a dedicated email address to receive complaints, capable of handling a large volume of reports in the standard Abuse Reporting Format. ( http://blog.returnpath.com/blog/jd-falk/arf-demystified )
4.) Select the "Sign in through Yahoo!" button below to start.

And a place where you can contact Yahoo help to indicate the IP addresses you are using for sending:
https://io.help.yahoo.com/contact/index?y=PROD_MAIL_ML&page=contactform&locale=en_US&selectedChannel=email-icon&yid=

Has anyone else had to go through this in the last 6 months or so? (from April 2015 and before)?

Thanks,
-Brett
http://www.bgswebdesign.com/Contact-Us.php

*** I do custom List Mail Pro installations ***
Contact me through my website (above)

BGSWebDesign

  • Posts: 625
    • View Profile
    • http://www.bgswebdesign.com
Re: Delayed sending and Deferred?
« Reply #8 on: April 28, 2015, 07:56:02 pm »
Hi DW,

Do you have some code that could handle the updated Yahoo FBL's?  Here's the specs (it's ARF format), and can be sent to any email. So, I'm thinking in Dailymail, when I handle the bounces now (as I do bounces to an email address), that I could insert some code to handle the FBL from Yahoo and delete the complaints, does that make sense?

Here's the link:
http://feedbackloop.yahoo.net/

Here's more from Yahoo central on it:
https://help.yahoo.com/kb/complaint-feedback-loop-program-sln3438.html

Bulk Sending standards:
https://help.yahoo.com/kb/recommended-guidelines-bulk-mail-senders-postmasters-sln3435.html

Help Central at Yahoo:
https://help.yahoo.com/kb/helpcentral

« Last Edit: April 28, 2015, 07:58:20 pm by BGSWebDesign »
Thanks,
-Brett
http://www.bgswebdesign.com/Contact-Us.php

*** I do custom List Mail Pro installations ***
Contact me through my website (above)

BGSWebDesign

  • Posts: 625
    • View Profile
    • http://www.bgswebdesign.com
Re: Delayed sending and Deferred?
« Reply #9 on: June 25, 2015, 01:46:08 pm »
Hi,

It seems there is no one replying here, so I thought I should chime in as it appears this is all moving over to Yahoo Customer Service on June 29, 2015... 

I've gone through the entire Yahoo Feedback Loop (YFBL) process.  It seems this requires setting up a few email address on your server, such as postmaster (or abuse), and then one other email address of your choice for handling the yahoo feedback emails that will be sent to you in ARF format. 

I've set this all up, it only took about 30 minutes, and requires several confirmations, and clicking through to verify the emails, as well as checking and knowing your DKIM signature 1.) domain, 2.) s= selector, or you can use * for all, 3.) send feedback to - the email you setup on your own for handling yahoo feedback, and 4.) your postmaster, or abuse email address at your domain.

It's all setup and now waiting for Yahoo! activation....  so, hopefully within 24 hours or so, I will have notification...  Here's what I find on the Yahoo website about the ARF file it will send for every complaint from a Yahoo user:
--
Each report consists of one message – with full headers, nothing “munged” or redacted – and some additional metadata as defined in the Abuse Feedback Reporting Format (ARF - http://en.wikipedia.org/wiki/Abuse_Reporting_Format).
The From: header will be ‘Yahoo! Mail AntiSpam Feedback <feedback@arf.mail.yahoo.com>', while the SMTP MAIL FROM (envelope sender) is of the format ‘cfl.<random>-<recipient address>=<recipient domain>@returns.bulk.yahoo.com’.
The source IP address may vary, but every report is signed with DomainKeys or DKIM using the signing domain ‘arf.mail.yahoo.com’. That signing domain is not used for any other purpose.
We cannot provide a sample report customized for your particular domain, but there is a sample in the ARF specification
--

What to do with the reports? ( from Yahoo: http://feedbackloop.yahoo.net/faq.php ):
Yahoo!'s expectation is you will take steps to reduce the number of complaints. There are many different ways to achieve this, depending on what kind of business you operate and what kinds of mail you transmit. Some of the most common techniques include:

    ISPs will likely figure out which of their users sent the message, determine if that user violated their policies, and then react appropriately – perhaps by sending a warning to the user, reducing their privileges, and/or removing them from the system.
    A bulk sender or mailing list operator could:
        choose to immediately cease sending additional similar mail to that recipient;
        review how that address was initially added to their list, and adjust their processes so that the same mistake is not repeated, perhaps even removing all recipients obtained through that mechanism;
        attempt to determine if the content of the message led to the report;
        attempt to determine if the message should have been expected by the recipient – for example, do the recipients have a clear understanding of the frequency and content of the list?

Or, contact Return Path for assistance in developing the right process for your company.
------------

Now, the next step - DW - do you have some code that could process the ARF files to remove these email addresses automatically, or should I write something along the lines of what is already there (in dailymail.php) to handle bounces by looking at this email address and removing these emails from all of my Listmail PRO lists every day when dailymail runs?

« Last Edit: June 25, 2015, 01:53:21 pm by BGSWebDesign »
Thanks,
-Brett
http://www.bgswebdesign.com/Contact-Us.php

*** I do custom List Mail Pro installations ***
Contact me through my website (above)

BGSWebDesign

  • Posts: 625
    • View Profile
    • http://www.bgswebdesign.com
Re: Delayed sending and Deferred?
« Reply #10 on: July 17, 2015, 07:42:54 am »
Hi Everyone,

This is to document what I've found with Yahoo Complaint Feedback loops (CFBLs).  Since my last post, Yahoo is now handling their own CFBLs (instead of using Return Path which I think they were using previously). 

From the looks of things, you have to re-submit your CFBL requests again, or at least try it, just to be sure it is still active.  You can do that at this page:
https://help.yahoo.com/kb/postmaster/SLN3438.html?impressions=true
(click on the option to Fill Out a New CFL Application)

The application requires that you have a postmaster@yourdomain.com email address, as they will send you a verification code immediately to that address. Which you must confirm before you can submit the application form.

There is more information on the transition over to Yahoo CFBL processing, versus Return Path and how to use them here:
https://help.yahoo.com/kb/postmaster/learn-complaint-feedback-loop-program-sln26007.html

(including this part)
-----------------------
If I didn’t save my existing CFL info from the Return Path site is there still a way to get it?
You'll need to contact Yahoo Customer Care to complete your request.
    Visit postmaster.yahoo.com.
    Click Contact Us | select Product Support.
    Enter your info in the required fields.
    Click Create Request.
To request CFL data for your domains, submit a New Sender Application.
Note: Yahoo only shares the CFL information with authorized users.
-----------------------

I've now gone through that step, which indicates that it will be 48 hours before I receive a reply.  I don't think my previous request was received, from the looks of things, and maybe this was due to them moving over the services to their own company. 

I will keep you posted on what I find...

DW - I'm still looking for the 'best practice' for handling the Yahoo FBL emails, I assume for now that I should process them with Dailymail as they will be coming in through an email address setup for yahoo complaints, is there some code you have for handling ARF format feedback to email that I can use?

« Last Edit: July 17, 2015, 07:48:27 am by BGSWebDesign »
Thanks,
-Brett
http://www.bgswebdesign.com/Contact-Us.php

*** I do custom List Mail Pro installations ***
Contact me through my website (above)

BGSWebDesign

  • Posts: 625
    • View Profile
    • http://www.bgswebdesign.com
Re: Delayed sending and Deferred?
« Reply #11 on: July 17, 2015, 07:56:48 am »
Hi All,

See my previous post/reply above this one for more information.

This reply is to document that I've also found another place with excellent information on the new Yahoo Complaint Feedback Loops (CFBLs), here:
https://help.yahoo.com/kb/postmaster/smtp-error-code-table-sln23996.html

This page has an excellent description of all Yahoo SMTP Error Codes and their meaning. They also include information on looking up your IP address on Spamhaus and other services to make sure you're not listed. 

You will also want to take note of the link on that page that says:  Submit your sending IPs for review.
That page seems to be a page that lets you submit your IP for review, so that Yahoo knows that you are sending large email volumes from that IP. I would recommend that you also submit your IP addresses there - as well as filling out the Yahoo Complaint Feedback Loop (as indicated above in the previous post).

The Spamhaus lookup page is at:
http://www.spamhaus.org/lookup/

I hope you're all doing well in making sure that your Yahoo CFBLs are in place, as it appears that all email being sent to yahoo.com email addresses is being rejected unless you have the proper CFBL in place.

See my last post - in this thread for more information...

DW - I'm still looking for the 'best practice' for handling the Yahoo FBL emails, I assume for now that I should process them with Dailymail as they will be coming in through an email address setup for yahoo complaints, is there some code you have for handling ARF format feedback to email that I can use?

« Last Edit: July 17, 2015, 08:01:39 am by BGSWebDesign »
Thanks,
-Brett
http://www.bgswebdesign.com/Contact-Us.php

*** I do custom List Mail Pro installations ***
Contact me through my website (above)

BGSWebDesign

  • Posts: 625
    • View Profile
    • http://www.bgswebdesign.com
Re: Delayed sending and Deferred?
« Reply #12 on: December 11, 2015, 08:20:54 am »
Still no replies here?

I'm once again faced with the problems integrating Yahoo's feedback loop into ListMailPRO.  It's going to require a custom Dailymail routine that will also check the Yahoo Feedback loop inbox, and remove those users as HARD BOUNCES.   DW, do you have any code to handle this?

Recently - I found I'm still receiving tons of deferred messages from Yahoo... a bit ridiculous considering I've been sending email to yahoo addresses for nearly 20 years now!

Here's some information about it from Yahoo:
https://help.yahoo.com/kb/postmaster/recommended-guidelines-bulk-mail-senders-postmasters-sln3435.html
and:
https://help.yahoo.com/kb/postmaster/complaint-feedback-loop-program-sln3438.html
and:
https://help.yahoo.com/kb/postmaster/yahoo-complaint-feedback-loop-service-faq-sln26007.html

Thanks,
-Brett
http://www.bgswebdesign.com/Contact-Us.php

*** I do custom List Mail Pro installations ***
Contact me through my website (above)

BGSWebDesign

  • Posts: 625
    • View Profile
    • http://www.bgswebdesign.com
Re: Delayed sending and Deferred?
« Reply #13 on: August 05, 2016, 11:04:37 am »
DW,

DW, do you have any code to handle this?  (Yahoo Feedback Loop)?

TIA...

Thanks,
-Brett
http://www.bgswebdesign.com/Contact-Us.php

*** I do custom List Mail Pro installations ***
Contact me through my website (above)