[Fix] Duplicate UIDs on Import - v1.8x

[gluntz]

We are seeing an increase in the number of users who, when attempting to unsubscribe, tell us either:

a.  Their record does not exist
b.  It brings up someone else's email address

I took a specific request and logged into phpAdmin to query the lm_users table.  It highlighted what seems to be the origin of the problem, but I don't know why or how it can be fixed. Seems to be a bug in ListMailPro.  Here is the scenario:

1.  I entered ListMail Admin and in this case found the record, but it showed as Removed (strange, because the user recev ied an email from us and tried to unsub. She retrieved a record for another user, so did not continue).

2.  When I queried on the the user's email addres in phpMyAdmin, I retrieved a record that showed the correct person, name, etc.  So far, so good.

3. However, I looked at the email source to get the UID, since that is the key for the unsub request.  It returned UID=83a4429.

4. So, I queried lm_users on that field and guess what?  It retrieed ANOTHER user's record(s). (Actually, it found four records with that UID - same person, why??)

In essence, it appears the root of the problem is in the creation of the email. We use the standard !remove field, and yet it is adding an incorrect UID. Hence the erratic results.

ONE ITEM TO NOTE:

We never had this problem until some 4 - 8 weeks back.  This was AFTER we installed the latest version AND we use the feature to combine lists to do a mailing (that is, setting up Queries). So - is it possible the bug lies in this area???

Please advise ASAP.

George

[BGSWebDesign]

Hi,

So, I queried lm_users on that field and guess what? It retrieed ANOTHER user's record(s). (Actually, it found four records with that UID - same person, why??)

This is very troubling.  I am currently using the previous version which does NOT allow query building, one of the advantages to me moving to the new version which I plan on doing soon is using the User Select (query builder).

If this is the cause of the problem I'd sure like to know, DW, do you have any ideas as to why this is happening?  If it is query related please keep us informed on when you may have a fix for this?  I suggest you login to this users account if you need to duplicate the problem....

Good luck and please let us all know what you find the solution is for this problem...

[DW]

Thanks for the excellent info - I hope to fix this immediately!

Okay, so the problem is in creation of the email and/or processing of message codes.

I have a few questions:  How large is the list you are mailing?  Do the same email addresses sometimes appear on many lists?  Do you insert any users with a custom script which might create duplicate UIDs?  Are you using SMTP for sending or internal PHP mail()?

I need to reproduce the symptoms to be able to provide a fix.  Do you have any idea how to start this happening on a clean install.  Probably not, huh - that's the tough one for me to figure out...  

Regards

[BGSWebDesign]

DW,

You should probably contact George off-list ASAP, get all of the info directly and get this sorted out - let us ALL know up here by posting when you have it fixed, as I won't be able to move over to the new version until I know 100% for sure that this is fixed...  

[DW]

George would you be willing to submit your server info so I can look at this on your installation?  The answers to my last post's questions would also be useful.  Whether I hear from you or not I will see if I can recreate the problem in the next day or two.

Regards

[BGSWebDesign]

DW,

Please let us know ASAP what you find out about this... I will not be able to run v. 1.8 knowing that this could be a huge problem.   I appreciate the work you've done on LMP and this Query feature is one of the very best, to not know if it works properly would cause huge problems for me in how I email my list.

[gluntz]

"I have a few questions: How large is the list you are mailing? Do the same email addresses sometimes appear on many lists? Do you insert any users with a custom script which might create duplicate UIDs? Are you using SMTP for sending or internal PHP mail()?"

Dean:

1. We are combining lists that range from a few hundred names to 50,000 names.  This particular one had 3 or 4 lists, largest of which was 4,700 names.
2. Yes - emails do appear in multiple lists, and we do select the option to cull duplicates.
3. No custom script - standard !remove.
4. We are using internal PHP mail().

I submitted our site info for you to use - with discretion and security, of course.

George

[DW]

George,

Ultimately, I'm going to have to recreate the error on my own, without using your server, but your answers do help a great deal - thanks.  I have one further question in response to your answer #3 - I was inquiring as to whether you had INSERTed users with a custom script (or ListMail plugin).  I'm not sure why you refer to the !remove code in your reply - you probably read the question wrong.

One more question, you state that you select the option to cull duplicates.  I assume you are speaking about the "filter duplicates" option in User Selection.  The user selection feature could have something to do with this.  Since I have access to your installation, can you tell me the names of some of your saved selections that you have emailed and run into this problem with?

Regards

[gluntz]

Dean,

1.  Yep - I mean "filter duplicates"  

2.  Never used scripts to add identities.

3.  Selections that we have used and where we know the symptoms have shown up are:

UTI/Candidate Customers
ADD/ADHD Special List

Hope this helps!

George

[DW]

Testing  

[DW]

Okay, after some useless tests with rem.php I considered your original post more thoroughly.  You state that many users have the same UID.  I logged into your PhpMyAdmin and verified that this is indeed true.  I am confident that this is the problem.  It would appear that ListMail is creating users with non-unique IDs.  I'm going to run some tests to see why this might be happening.

As for a fix for current users, I can write a script that will go through your database and change duplicate UIDs to a unique one.  Your users whose UID has changed will notice upon the next mailing (and their remove links will work).

Regards

[DW]

This MySQL command displaying duplicate IDs and the counts for each ID appears to be the best I can get:

SELECT count(*) AS cnt FROM `lm_users` WHERE 1 group by uid order by cnt desc

One UID is duplicated 57 times :?

I'll work on that script to fix your users for you now.  If I can recreate the problem importing large lists on my own test installation I will issue a ListMail update.

Regards

[DW]

Please stand by...  The problem has been confirmed on my own test installation - it seems to affect everyone.  I'm going to have to issue a database update that might take awhile for those with many users.. :x

I might run a test to see if this problem also plagued v1.77 members, as the process was practically the same..  Perhaps there were simply more collisions in v1.81 so it was more noticeable.

I am working on it - I will issue a general update ASAP!

[DW]

I have verified that this is a v1.8x-specific bug.  The bug only affects imported users.  It came about when I added the "signup-xtra.php" capability to the import process along with the welcome and confirmation emails.  To use signup-xtra.php I needed to define the MySQL link variable in MySQL-specific functions to keep the script working in case signup-xtra.php contained additional database connection routines. The problem was that I defined the MySQL link variable in the mysql_num_rows() PHP function used to determine if a UID already exists.  This function does not require the MySQL link definition and adding it resulted in an unexpected response.  As a result, ListMail was not detecting any existing UIDs to ensure that the created UID was unique.

Now that the problem is solved and I have optimized UID creation I will be working on the upgrade/fix.

Regards

[DW]

Alright!  I have released v1.82 as a fix for this issue.  Phew!  

Regards