Mark mail addresses from LDAP as verified

W've noticed some time ago that the mail adresses of some/most/all users created from LDAP don't have verified mail addresses which makes Postorius unusable. There was an attempt in the past to fix this with 4e92f802. I remember that the fix seemed to work, but I never verified it. Due to an independent incident, all Postorius users were deleted and recreated a few weeks ago. Now almost no user has a verified mail address.

As a workaround users can manually verify their mail address.

In the Django admin interface it looks like the EmailAddress objects that 4e92f802 marks as verified only exist for very few users. All users have the correct mail address in their User object but hat object does not have a verified flag. Also all the list subscriptions look good for all users.

assigned to @julian

I did some testing:

• Creating a new Postorius/Hyperkitty account (via OAuth2 login or ldap_backend.populate_user(user).save()) does not create a EmailAddress object for the account.
• Opening https://lists.cccv.de/accounts/email/ creates an EmailAddress object for User.email with the verified and primary attributes set to False.
• Deleting an account in Postorius/Hyperkitty unsubscribes the mail address in OAuth2/LDAP (regardless of the value of User.email or which EmailAddress object is marked as primary) and all verified mail addresses from all mailing lists.
• Adding a mail address on https://lists.cccv.de/accounts/email/ and making it primary cases User.email to be set to this address. It is reset by the next login, probably also by the next LDAP sync. After User.email is reset, the "Primary email" on https://lists.cccv.de/user-profile/ is the value of User.email (i.e. the value from OAuth2/LDAP) while https://lists.cccv.de/accounts/email/ shows the mail address manually made primary as the primary address. User.email can again be changed by the user to any unverified address.
• Marking an EmailAddress object as verified does not have any immediate effect on the list roles in Postorius. Opening https://lists.cccv.de/user-profile/ and then going back to the overview however makes the lists appear.
• Unverifying an EmailAddress object in Django-Admin again immediatly makes the list roles disappear in Postorius. Marking it as verified again does not have an effect until https://lists.cccv.de/user-profile/ is opened.
• Hyperkitty seems to ignore EmailAddress objects entirely. It also seems to do some level of caching which makes everything more annoying. I honestly fail to understand what Hyperkitty does exactly. So I guess it time to read through it's code.

One thing I wan't fully aware of is that Mailman, the core compontent that runs as an independent Django application, has it's own user database. Mailman users are identified by their mailman user id (a hex-encoded sequence of 16 random bytes) and can have any number of mail addresses.

Hyperkitty uses django_mailman3.lib.mailman.get_subscriptions to get the lists a Django user is subscribed to. The function internally resolves the Mailman user id/object based on User.email. It uses Django's caching subsystem to cache the Django user id -> Mailman user id relationship indefinitly. Since we did not setup any caching backend, it caches in memory, so restarting mailman-web clears the cache.

Postorius uses the mail addresses of the verified EmailAddress objects associated with the Django user object to get list roles/memberships by mail address.

I previously observed that visiting the user profile was neccessary to see changes in Postorius. This is not the case. Instead Postorius behaves completely different if the index page is visited with "All" selected vs. any other filter option. With "All" it only shows non-advertised lists to admins and no one else (not even members!). With any other filter option it filters by actual membership/role. Visiting the profile and then clicking the navbar link took me to the "Filter by Role" page while I previously had the "All" page open. I didn't notice that because the dropdown says "Filter by Role" in both cases.

Edit: Oh and I had the "All" page open, because Postorius redirect to it if the logged-in user is not a member of any list.

• Mailman Web's Datamodel: Users have an email attribute and any number of verified/non-verified EmailAddress objects. List roles are only stored in Mailman's database.
• Mailman's Datamodel: Users have any number of mail addresses. User mail addresses are members/moderators/owners of lists.

If an address is added to a list in Postorius that does not exist in Mailman, a new Mailman user is created without a corresponding Mailman Web user. If this address is later added to a Mailman Web user, the Mailman users need to be merged. Postorius does this if the address is added and verified via the web interface. We need to take care of this in the sync. Otherwise Hyperkitty will only display the lists of one of the Mailman users.

Making an unverified mail address the primary address and implicitly setting User.email to this unverified address is an edge case in allauth (the module that provides the EmailAddress models). If a user does not have any verified addresses, an unverified address can be set as the primary address. Additionally primary addresses cannot be deleted.

So if we make sure that all users have at least one primary mail address that is verified, it should no longer be possible to make an unverified mail address the primary address or to set User.email to an unverified address.

This is not possible with the current split implementation, so I plan to merge https://git.cccv.de/uffd/django-auth-ldap-remoteuser into this module.

closed with commit 50548dc9