Author Topic: Sending to AOL, HotMail, Yahoo and other free services  (Read 61292 times)

DW

  • Administrator
  • Posts: 3787
    • View Profile
    • https://legacy.listmailpro.com
Sending to AOL, HotMail, Yahoo and other free services
« Reply #30 on: June 09, 2007, 02:23:06 am »
Brett,

Email addresses in message headers, and possibly the body, are indeed filtered (not sure why they do this).  The AOL complainant removal script  checks for the remove link, which is not removed from each returned message.  ARF is fine - they still return the original message in it's entirety including the remove link.

Regards
Dean Wiebe
ListMailPRO Author & Developer - Help | Support | Hosting

DW

  • Administrator
  • Posts: 3787
    • View Profile
    • https://legacy.listmailpro.com
Sending to AOL, HotMail, Yahoo and other free services
« Reply #31 on: June 15, 2007, 02:10:30 am »
The AOL auto-removal code has been updated to handle more types of remove links.
Dean Wiebe
ListMailPRO Author & Developer - Help | Support | Hosting

BGSWebDesign

  • Posts: 625
    • View Profile
    • http://www.bgswebdesign.com
Sending to AOL, HotMail, Yahoo and other free services
« Reply #32 on: June 16, 2007, 08:38:46 pm »
DW,

Please check your code, this new code did NOT work for me, even though the old code worked just fine!  I tested it many ways, but I can tell you this, the old code runs fine and takes about 5-6 seconds to process, the new code runs immediately (1 second), reports no errors, and does NOT send an email notifying me, nor does it remove the invalid User ID.  

Another thing to note, when running the test, you MUST find the UserID (UID field), and NOT the ID field.  Here is how you test it.  You must create a test file, call it 'aoltest.txt', mine is below:
Code: [Select]


From: <scomp@aol.net>

http://mydomain.com/mail/rem.php?u=ef9b530



Make sure you replace mydomain.com with your own domain, AND that you go into PhpMyAdmin and locate a test email address you want to remove - after locating a test address, do a browse and find that address in your MySQL database, then replace the 'ef9b530' shown above with the UID column that shows in your MySQL database for the user you want to do a 'test remove' - make sure that user has a CNF field of '1' (active) - otherwise the remove will not work and report an error (to your email address) that the user is not in the database.

Then, save it as ASCII text file and upload it to the folder you will place the aolnotify.pl script (shown below).  Then Login to your shell account and enter this at the command prompt (from the folder where you have placed the aolnotify.pl file):
Code: [Select]

./aol-notify.pl < ./aoltest.txt


You should receive an email address in the address you specified indicating that the user has been removed (see below - replace aolnotify\@mydomain.com with the email address you want to be notified at at your own domain.  You can also verify that it removed the address by reloading the MySQL database in PhpMyAdmin and looking for the email address that you used (in aoltest.txt), after you locate it, the CNF field should be changed to '2' if that address was successfully removed.

For anyone looking, here's the 'old code' (aolnotify.pl) that I used that worked:
Code: [Select]

#!/usr/bin/perl -w
# config
$my_email = "aolnotify\@mydomain.com";

# get piped message
$em = '';
$hd = '';
while($l=<>){
 $em = $em . $l;
 if((index($l,'From: ')!=-1 && index($l,'scomp@aol.net')==-1) || (index($l,'To: ')!=-1 && index($l,'Delivered-To:')==-1) || index($l,'Subject: ')!=-1){
  $hd = $hd . $l;
 }
 if(index($l,'rem.php?')!=-1){
  # print "l [$x]: $l"."l [$x]: 0123456789012345678901234567890123456789012345678901234567890123456789\n";
  $rl = $l;
  $rl = substr $rl, index($rl,'http://');
  $epos = index($rl,'"');
  if($epos==-1){ $epos = index($rl,' '); }
  if($epos==-1){ $epos = index($rl,"\n"); }
  if($epos==-1){ print "eol not found.."; return 1; }
  $rl = substr $rl, 0, $epos;
  $rl =~ s/=3D/=/g;
  $rl =~ s/\&/\\\&/g;
 }
}
$qstr = "\nFrom: <scomp\@aol.net>";
$apos = index($em,$qstr);

if($apos!=-1){
 if(!$rl){
  #print "AOL spam complaint but NO REMOVE LINK... $hd";
  exit;
 }
 #print "AOL SPAM COMPLAINT - Remove Link FOUND! $rl\n";
 sub rsgen{
  my $rslen=shift;# the length of
  my @ch=('a'..'z','A'..'Z','0'..'9','_');
  my $rs;
  foreach (1..$rslen){ $rs.=$ch[rand @ch]; }
  return $rs;
 }
 my $rs=&rsgen(8);
 $cmd = "curl -o /tmp/aol$rs $rl\\&dodel=1 \> /dev/null 2\> /dev/null";
 #print "\ncmd=$cmd\n";
 system($cmd);
 sleep(5);
 $cmd = "mail -s \"AOL Complaint Processed\" $my_email < /tmp/aol$rs";
 system($cmd);
 system("rm -rf /tmp/aol$rs");
}



After you test it, report if it works, than try it with the new version DW has updated, does it work for that version too?  I can't get the new version to work for me...  

Any ideas DW?
Thanks,
-Brett
http://www.bgswebdesign.com/Contact-Us.php

*** I do custom List Mail Pro installations ***
Contact me through my website (above)

DW

  • Administrator
  • Posts: 3787
    • View Profile
    • https://legacy.listmailpro.com
Sending to AOL, HotMail, Yahoo and other free services
« Reply #33 on: June 18, 2007, 02:28:41 am »
Brett, thanks for reporting this (and for your very helpful response) - the script was totally broken.  Indeed, I just grabbed what I thought was my latest revision and didn't test it. :oops: I have again updated the code - let me know if you have further troubles.
Dean Wiebe
ListMailPRO Author & Developer - Help | Support | Hosting

BGSWebDesign

  • Posts: 625
    • View Profile
    • http://www.bgswebdesign.com
Sending to AOL, HotMail, Yahoo and other free services
« Reply #34 on: June 19, 2007, 08:45:43 am »
DW,

Quote
I have again updated the code - let me know if you have further troubles.


That still doesn't work?  I get the same response, it returns immediately (< 1 second) and it doesn't set the CNF field in the database, it also does not send me any notfication.

Also, another problem.... the original code you posted (which I posted above) works when I run it from the command line, but I'm now getting emails from AOL and it doesn't appear to be working because I click the 'REMOVE' links in the attached messages I receive from AOL and these users are not removed from the database!

Shouldn't the feedback loop remove them from ALL lists immediately when it runs?

Here is a paste in of one of my messages I receive from AOL:
Code: [Select]
This is an email abuse report for an email message received from IP address
xxx.xx.xx.xxx on Tue, 19 Jun 2007 08:06:04 -0400

For information about this format please see
http://www.mipassoc.org/arf/

For information about AOL E-mail guidelines, please see
http://postmaster.aol.com/guidelines/

If you would like to cancel or change the configuration for your FBL please
contact our postmaster help desk. Up to date contact information for the
postmaster help desk is located at
http://postmaster.aol.com/contact



--------------------------------------------------------------------------------
Attachment: message/feedback-report (0.2K)


Display / Download
 


The message also contains an attachment, which I downloaded (appears below) followed by what looks like the entire original message, here's the attachment:
------------
Code: [Select]
Feedback-Type: abuse
User-Agent: AOL SComp
Version: 0.1
Received-Date: Tue, 19 Jun 2007 08:06:04 -0400
Source-IP: xxx.xx.xx.xxx
Reported-Domain: server.mydomain.com
Redacted-Address: redacted
Redacted-Address: redacted@



The entire message then appears after this - the message appears to include the correct header info, and the end of the message with the removal code, here's a paste in of most of the message:
------

Code: [Select]
From:  "My Domain" <brett@mydomain.com>
To:  redacted@aol.com
Date:  19 Jun 2007, 08:05:08 AM  
Subject:  Holleigh your *new Destiny Astrology Reading...  
           
 

--------------------------------------------------------------------------------

Hi Holleigh,

It's been a really crazy year, so I Had to email you
right away and ask you:

Do you want to have a better idea of what lies ahead for
you?

MESSAGE TEXT REMOVED.....

To be canceled from this mailing, choose the link below:
http://mydomain.com/mail/rem.php?u=8addu8v

To be canceled from ALL mailings at My Domain, choose:
http://mydomain.com/mail/rem.php?c=13&u=8addu8v


Can you tell me what is going on?  Will you provide a fix for this, for now it looks like I'm going to have to go through hundreds of messages every time this comes in and remove the users by hand, unless there is some other way to remove these users?

I keep getting this sent to my email notify address:
Code: [Select]
<html>
<head>
<title>Email Address Not Found!</title>
</head>
<body bgcolor=white>
Your email address (!email) was not found in our database!  This error should
never happen, please contact the administrator of this web site for a manual
removal.<br>
</body>
</html>


I've uncommented the print statements and the actual remove link looks fine, here's what I see from the shell:
Code: [Select]
./aolnotify.pl < ./entiremessage.txt
RL=http://mydomain.com/mail/rem.php?c=13&u=8addu8v
APOS,BPOS=-1,638
AOL COMPLAINT - Link Found! http://mydomain.com/mail/rem.php?c=13&u=8addu8v
CMD=curl -o /tmp/aolMoBhnb_p http://mydomain.com/mail/rem.php?c=13&u=8addu8v&dodel=1 > /dev/null 2> /dev/null


Do you know why this is not working???  It's strange because I can run the exact command (found above in the curl statement on the URL of my browser and it works)?  I also tried replacing the curl command with wget and it still doesn't work???
Thanks,
-Brett
http://www.bgswebdesign.com/Contact-Us.php

*** I do custom List Mail Pro installations ***
Contact me through my website (above)

DW

  • Administrator
  • Posts: 3787
    • View Profile
    • https://legacy.listmailpro.com
Sending to AOL, HotMail, Yahoo and other free services
« Reply #35 on: June 20, 2007, 03:04:10 am »
Brett,
Quote
It's strange because I can run the exact command (found above in the curl statement on the URL of my browser and it works)? I also tried replacing the curl command with wget and it still doesn't work???

What happens when you run the full curl command from an SSH shell?  (You can skip the 1> /dev/null 2> /dev/null part for more info)

I'm thinking that the solution is either to replace ? with \? and & with \& or, I believe, surround the URL in double quotes, as I have done in the latest code...

Also, I may have messed this part up in the latest revision:
Code: [Select]
$apos = index($em,"\nFrom: <scomp>");
I am waiting for a complaint on a test server to double-check this.  I've changed it to this just in case:
Code: [Select]
$apos = index($em,"\nFrom: <scomp");
Quote
Shouldn't the feedback loop remove them from ALL lists immediately when it runs?

Not necessarily - the script simply follows whichever remove link is found in the message.

Regards
Dean Wiebe
ListMailPRO Author & Developer - Help | Support | Hosting

BGSWebDesign

  • Posts: 625
    • View Profile
    • http://www.bgswebdesign.com
Sending to AOL, HotMail, Yahoo and other free services
« Reply #36 on: June 20, 2007, 12:36:28 pm »
DW,

Quote
What happens when you run the full curl command from an SSH shell?


It acts like it runs and reports that it is calling the page, but then when I look in the database the CNF field has NOT been changed to '2', instead it is still '1' - so I don't have a clue what is going on here?

Quote
am waiting for a complaint on a test server to double-check this.


I've been debugging it myself and spent a few hours on it yesterday, what I did was just saved the entire message that AOL sent to me and I use that for testing.

You're right about the <scomp you had it right the second time, you have to check it both WITH and WITHOUT the angle brackets as the mail I'm getting now from AOL is without the angle brackets so it's like this:
Code: [Select]

From: scomp@aol.net


Quote
with \? and & with \& or


Yes I tracked down that problem too - I had to removed the extra \ in front of the &, so it's like this:
Code: [Select]

#  $rl =~ s/\&/\\\&/g;


As you see I commented out that line.

But, either way, even when I get the CORRECT SYNTAX and call the thing with CURL or with WGET - either way it does not work... so, I had a thought last night anyway - there's no reason to call this out on the command like that - it would be better anyway to just include the lmdelete.pl file and then call something similar to the lmdelete function but to call it with the UID field instead of with email - so, instead of calling it like it is called now:
Code: [Select]

LMdelete('9,10,11,13,14,15,16,24,27,28,29,32',$lmpemail);


Writing and calling a different function I'll call it LMdetelALL with the UID like tihs:
Code: [Select]

LMdeleteALL($lmpuid);


So, I'm off to look at that, if you have time, and add the function let me know, I might not have time to get to it by tonight as I'm busy on something else.   Let me know - but I think this sounds like a better way to do it - there's too much to worry about with calling WGET or CURL.

What do you think?
Thanks,
-Brett
http://www.bgswebdesign.com/Contact-Us.php

*** I do custom List Mail Pro installations ***
Contact me through my website (above)

DW

  • Administrator
  • Posts: 3787
    • View Profile
    • https://legacy.listmailpro.com
Sending to AOL, HotMail, Yahoo and other free services
« Reply #37 on: June 24, 2007, 06:38:05 pm »
I will consider how to better improve this script such as by removing the user from all lists.  (A simple GET var added to the command-line by the AOL script could tell a modified rem.php to remove from all lists).  Considering how infrequent complaints will be I think wget/curl should work fine - I don't see a need to go to a Perl method.

For now I have updated the code again with one important change to avoid the problem of varying "From" and "Return-path" headers.  (I also noticed a serious bug in the last post....)

The search line is now as follows:
Code: [Select]
$apos = index($em,"scomp\@aol\.net");
Regards
Dean Wiebe
ListMailPRO Author & Developer - Help | Support | Hosting

BGSWebDesign

  • Posts: 625
    • View Profile
    • http://www.bgswebdesign.com
Sending to AOL, HotMail, Yahoo and other free services
« Reply #38 on: July 02, 2007, 12:16:34 pm »
DW,

Quote
For now I have updated the code again


Congrats, it works fine now.  I did have to add in some of my own code to parse out another type of mailing I do, but all works now.
Thanks,
-Brett
http://www.bgswebdesign.com/Contact-Us.php

*** I do custom List Mail Pro installations ***
Contact me through my website (above)