Sending to AOL, HotMail, Yahoo and other free services

[DW]

It is recommended that anyone sending large amounts of email to AOL apply to become whitelisted, set up a feedback loop to be notified of spam complaints, and adhere to the guidelines.

http://postmaster.info.aol.com/tools/whitelist_guides.html

I have created a script that can be used to process AOL spam complaints to ListMail-sent email.  It will basically "click" and confirm the remove link in your message automatically to remove the user.  I will reply to this post with the (Perl) script in case you want to use it.

HotMail's official Postmaster site:
http://postmaster.msn.com/

Some interesting discussion about HotMail blocking:
http://www.webhostingtalk.com/showthread.php?threadid=224942
http://forums.devshed.com/showthread.php?t=109369

I am still continuing my efforts and have not fully analyzed Yahoo, gMail, or other free service providers.  Please post any information you have about email being blocked by free services!

Regards

[DW]

Set up an email alias/forwarder/pipe forwarding email sent to your AOL Feedback Email to this script, which should be saved with the .cgi extension and stored in your cgi-bin folder.

Updated: June 24, 2007

Code: [Select]

#!/usr/bin/perl -w

# hostname - shown in notifications
$hostname = 'MyServer';
# notify? 1 or 0
$notify = 1;
# email to notify
$my_email = 'test@example.com';

# get piped message
$em = '';
while($l=<>){
 # replace confirm links to allow removal
 $l =~ s/confirm.php/rem.php/gi;
 $em = $em . $l;
 if(index($l,'rem.php?')!=-1){
  $rl = substr $l, index($l,'http://');
  $epos = index($rl,'"');
  if($epos==-1){ $epos = index($rl,' '); }
  if($epos==-1){ $epos = index($rl,"\n"); }
  if($epos==-1){ print "eol not found.."; return 1; }
  $rl = substr $rl, 0, $epos;
 }
}

$apos = index($em,"scomp\@aol\.net");
if($apos!=-1){
 if(!$rl){ print "AOL COMPLAINT but NO REMOVE LINK"; exit; }
 print "AOL COMPLAINT - Link Found! $rl\n";

 #randgenerator
 sub rsgen{  my $rslen=shift; my @ch=('a'..'z','A'..'Z','0'..'9','_'); my $rs; foreach (1..$rslen){ $rs.=$ch[rand @ch]; } return $rs; }

 # follow remove link
 my $rs=&rsgen(8);
 $cmd = "curl -o /tmp/aol$rs \"$rl\&dodel=1\" 1\> /dev/null 2\> /dev/null";
 system($cmd);
 # notify
 if($notify==1){
  $cmd = "mail -s \"$hostname - AOL Spam Complaint\" $my_email < /tmp/aol$rs";
  system($cmd);
 }
 system("rm -rf /tmp/aol$rs");
}


[BGSWebDesign]

Hi DW,

I had a BIG problem today with AOL.... for some reason, today is the day I started getting spam reports sent to my shared host, note I've been sending email from a shared host for years!  AOL could not confirm the problem from my DOMAIN IP address, they said all is fine... after further checking it seems that it's the shared mail domain IP that is blacklisted with MY DOMAIN indicated to be blocked... so apparently all email blacklisted from this shared mail server - but BECAUSE of my domain sending...   after calling AOL back they confirmed the shared mail server was blacklisted.

I came up here looking for an answer.... now, my question regarding this, does this 'feedback loop' script REQUIRE a dedicated server?  Here's the info on the feedback loop at aol:

AOL offers a feedback loop service, free of charge, to parties sending large amounts of mail to AOL members. The feedback loop (FBL) will forward any mail reported as spam originating from the associated IP addresses back to the listed email address. We highly recommend the use of a dedicated e-mail address for this purpose.

So, I wonder what happens if I enter the shared mail server IP address in the IP address box for the feedback loop?  Does that mean that ALL bounced email from that shared IP is sent to my email address, or does AOL filter out the feedback by REPLY-TO first and only feedback the message to MY FEEDBACK Email address?

Do you know how this might work DW, or anyone???  Urgently need to answer this...

[DW]

Brett,

I host a number of clients running mailing lists and AOL, in my experience, has been very quick at responding to phone calls to their postmaster hotline.  I'm usually unblocked instantly and back on the whitelist shortly thereafter.  This may be in part due to the fact that I assure them I'm in control of the clients on my server and am (now) automatically processing feedback with the above-mentioned script to remove complaining users to my clients' lists.  HotMail and Yahoo can't seem to figure to offer a feedback loop, unfortunately..

So, I wonder what happens if I enter the shared mail server IP address in the IP address box for the feedback loop? Does that mean that ALL bounced email from that shared IP is sent to my email address, or does AOL filter out the feedback by REPLY-TO first and only feedback the message to MY FEEDBACK Email address?

I believe AOL can send reports to anyone regarding an IP.  I am not sure if you will receive reports for anyone mailing from the same IP.  What I recommend is:

• 1) Your host contact the AOL postmaster hotline by phone.  A service representative will run a check against your IP and possibly escalate you to another department to receive a phone call for further investigation.  The block that can be in place will show up in the /var/log/maillog file as a URL. (listed at http://postmaster.aol.com/errors/index.html)  This information is handy to have when contacting AOL. Here's a common error for excessive complaints and/or large amounts of email: http://postmaster.aol.com/errors/421dynt1.html
  
  
• 2) Your host should probably do this regardless and first.  Apply to be whitelisted at AOL and set up an email address to receive feedback email.  The application will be scanned and if rejected you might be instructed to phone the hotline.
  
  
• 3) You may be able to get your mailings/server unblocked by phoning and stating you are simply a client and need to get your email out and that you are contacting your host.  I think it would also be useful to mention that you can set up a feedback loop script (mentioned above, with an email alias similar to bounce.cgi) to automatically remove users who generate complaints against your mailings (which all contain "remove" links that simply need to be followed). You may also want to inquire about whether they can limit feedback based on your "bounce to" and "send from" addresses.  If they can't you might still benefit from the script above as you do not need to be notified of processed emails/complaints.  Give AOL a call at one of the numbers listed on this page.
  

Hope that helps!

Regards

[BGSWebDesign]

Hi DW,

Thanks for your excellent reply...  one question, you say:

The application will be scanned and if rejected you might be instructed to phone the hotline.

I have setup the feedback script as you recommended, and now have an aol_complaint email address forwarding to that script for removal...
1.) How do I test it?
2.) They unblocked me immediately, but say it may be up to 72 hours for White Listing to be turned on, how long did you find it normally takes for you to be scanned, and put into White List, my host has given me until Friday...

[DW]

I have setup the feedback script as you recommended, and now have an aol_complaint email address forwarding to that script for removal...
1.) How do I test it?

By default the script sends an email with the output of the HTML remove success page.

Code: [Select]

$cmd = "mail -s \"AOL Spam Complaint\" $my_email < /tmp/aol$random
_string1";
system($cmd);Set the $my_email variable to your email address.

2.) They unblocked me immediately, but say it may be up to 72 hours for White Listing to be turned on, how long did you find it normally takes for you to be scanned, and put into White List, my host has given me until Friday...

I'm afraid I can't be of much more help in this area.  I didn't pay specific attention to how long each of my whitelist requests took.  I've never heard of anything about scanning.  If your host is making an issue of this and pressuring you I would find it unfair as there are other users on the server who could be causing complaints, too.  Complaints are not uncommon even with completely opt-in lists as subscribers often forget they subscribed.

Regards

[mike2]

Having a problem with this script....  My feedback loop is my main email, so I  setup an email called aolcomplaint @ mydomain, when I get a complaint from aols loopback, I just forward it as a bounce to aolcomplaint @ mydomain...

This should work I believe, but here's what I get with ever email I've tried so far (about 5 or 6) and I have manually checked to see if the email address was in my database and it was there, so I'm not sure whats happening, any suggestions?

Here's what the message I get looks like, which basically says it's not finding the address, but it's looking for ( !email ), which doesn't seem to be breaking down to the users email:

Code: [Select]

<html>
<head>
<title>Email Address Not Found!</title>
</head>
<body bgcolor=white>
Your email address (!email) was not found in our database!  This error should never happen, please
contact the administrator of this web site for a manual removal.<br> </body> </html>

UPDATE:

I found what may be the problem for me, I'm not sure.  I stopped the script from deleting the temp files and noticed the one looks like this:  

Code: [Select]

<a href=3D"http://www.my-domain.com/m/rem.php?u=3D6cf977e">http://www.my-domain.com/m/rem.php?u=3D6cf977e</a>


I'm guessing it's supposed to be just one URL, not an HTML like statement.  
Let me know if this is the problem and how I'd fix it, thanks.

[DW]

Mike,

<a href=3D"http://www.my-domain.com/m/rem.php?u=3D6cf977e">http://www.my-domain.com/m/rem.php?u=3D6cf977e</a>

The script simply finds the remove URL and 'follows' it.  It would appear that the link is coming through incorrectly, possibly by your sending of HTML mail :?:  The "3D" should not be in the string.  I suspect this link would work, with the 3D removed:
http://www.my-domain.com/m/rem.php?u=6cf977e

We need to determine where the message is being malformed.  Do you also receive the AOL email in it also being processed by the script?  Can you send me what an (HTML?) remove link looks like in that message?

The 3D should actually be %3D, I believe, and the % is being lost somewhere.  If we can find the string WITH the % we should be able to do a PHP "urldecode" or something else to format the string/message properly.

Regards

[mike2]

I'm actually not sending HTML e-mail.  A long time ago I set an option to automatically add the HTML part for aol addresses and have never changed it.

What I send my message it looks like a normal link:
http://ma-enews.com/cgi-bin/to.cgi?l=maadop

But upon examining the email returned by AOL's feedback loop, every URL in my message looks like:
<a href=3D"http://ma-enews.com/cgi-bin/to.cgi?l=3Dmaadop">h=
ttp://ma-enews.com/cgi-bin/to.cgi?l=3Dmaadop</a>

So, I'm guessing this has to be Listmail adding this.  I know I really no longer need that option because most Newer AOL browsers actually make the hyperlinks work properly by default, but I can't seem to find that option anywhere anymore...

[BGSWebDesign]

Hi DW,

I'm also curious as to what Mike is experiencing, maybe I have the same problem?

Set the $my_email variable to your email address.

I've done that, and the script seems to be setup properly 755 permissions, etc... but I NEVER receive an Email to my email address - wouldn't I see this subject: "AOL Spam Complaint"?

I never get any... but AOL is not complaining and they have white listed me, and my host is not complaining either so I'm assuming the feedback loop must be working?

Still, I'd like some assurance it is... anything else I should try... can I run this script from the command line to test it?

How did Mike look at the local temp files?

[mike2]

webshaman,

I am receiving the email from the script just fine.  But my email is showing that the email address isn't getting removed due to my URL not being formatted properly.  (Which I believe is because of an older setting that I set a while back, but can't find out how to un-set.)

I looked at the /tmp files because I run my own server(s).  I doubt if you are in a shared environment you can do this (or at least you shouldn't be able to).  

But if you look at the script, that is where it stores the files (email) for processing.  You can probably change the directory to a temporary directory under your home directory and then stop the script from deleting the files.  You would/should then be able to see them.  

As for your problem, make sure the email address you set $my_email to is correct and you kept the "\" before the "@" sign else it won't work.

[BGSWebDesign]

Hi Mike, DW,

I think this needs to be looked at in detail...  what I found is that when copying/pasting the script from above it didn't come through correctly in my editor...

This line was not correct:

Code: [Select]

 $cmd = "mail -s \"AOL Spam Complaint\" $my_email < /tmp/aol$random_string1";
since the ramdom _string1 were separated by a space!

make sure the email address you set $my_email to is correct and you kept the

I have this setup correctly...  

Thanks for the tip Mike, I've completely updated the code to write the tmp files to another temp folder on my server - and NOT to delete them, I'll let you know if I now get the emails, and what the files are showing... but still the question for DW...

How would anyone test this without re-writing the script like I have done?  It doesn't seemt his script can be run from the command line - since it's looking for input...

[DW]

Set up a file test with the following in it, including the top line break:

Code: [Select]


From: <scomp@aol.net>

http://example.com/mail/rem.php?u=test123
Replace "example.com/mail" with your ListMail domain and path.

Now you can test the AOL script from an SSH shell with the following command.

./aol-notify.pl < ./test

Regards

[mike2]

Ok, I am still having a problem with this script for some reason...

I just updated to 1.85 because I thought that removing the undocumented URL feature to aol would fix my problem but it hasn't.

Now all I get back in my email from the script is this:

admin-60-Incorrect information in file: './maoptin/lm_chtmld.frm'

I looked at a raw view of the email I get back from aol and in every URL right after the equal sign is "3D", looks like this:

http://ma-enews.com/cgi-bin/to.cgi?l=3Dmaadop

Somehow somewhere things are getting converted to ascii or something, because when looking at the raw view of the message I see this:

=93The Short Book on the Offer in
Compromise and IRS Liens and Levies!=94

The =93 and =94 above were originally quotes ( " " ).

I am using Pegasus for my email program, could this be the problem?  I am thinking someone Pegasus is doing this and of course when I then bounce it to my aolcomplaint email address it's not right.

I know I could change my aol white list address, but it'd be easier to fix this problem (maybe), because I want to see the original message sometimes.

I am going to telnet into my server and read the email in pine next time to see if I can tell if pegasus is adding the 3D.  Otherwise I'll just need to remove the "3D" string from the URL in the script.

UPDATE:

Ok, when I bounce this mail to my yahoo email address or back to myself, the "3D" isn't there, now I am lost...

On a side note Dean, when testing a link tracking link before saving it, I get the same error:

admin-60-Incorrect information in file: './maoptin/lm_chtmld.frm'

Well, now I can't re-duplicate that error, but it showed up once when I first started setting up "Link tracking", which was just today after updating to 1.85.

[BGSWebDesign]

Hi Mike,

I know I could change my aol white list address, but it'd be easier to fix this problem (maybe), because I want to see the original message sometimes.

Did you try the test above from DW?

I don't believe mine is workign correctly either, and I have not had time to run the test as outlined...  but I'm not getting the proper bounced messages and I know my email address is setup correctly...

There might be some ASCII conversion going on, most likely it's in the browser, or your reader, you would need to ENCODE/DECODE when it comes off the command line to see it correctly - and that could be your problem... so it probably looks alright to the script, but that doesn't explain why I'm not getting any AOL bounce messages...  I'll run the test in the next day or so and let you know what I find... I suggest you try running the test too - from your command line...  let us know what you come up with?