uffd-ldapd issueshttps://git.cccv.de/uffd/uffd-ldapd/-/issues2022-04-20T21:30:23Zhttps://git.cccv.de/uffd/uffd-ldapd/-/issues/4CacheControl leaks memory with certain workloads2022-04-20T21:30:23ZJulianCacheControl leaks memory with certain workloadsWe noticed that each sync run of [our mailman-django plugin](https://git.cccv.de/uffd/postorius-ldap-membership-management) causes the uffd-ldapd process to leak a few MB of memory. After a few days of running every 15 minutes this amoun...We noticed that each sync run of [our mailman-django plugin](https://git.cccv.de/uffd/postorius-ldap-membership-management) causes the uffd-ldapd process to leak a few MB of memory. After a few days of running every 15 minutes this amounts to hundreds of MB.
I was able to reproduce this locally with a script written based on uffd-ldapd's log output during a sync run. I was also able to verify that this is related to CacheControl, since it only occurs if caching is enabled.
Test script:
```python
import ldap3
import gc
def get_conn():
server = ldap3.Server('ldap://127.0.0.1:1337', use_ssl=True)
conn = ldap3.Connection(server)
conn.bind()
return conn
conn = get_conn()
conn.search('ou=users,dc=cccv,dc=de', '(&(objectClass=posixAccount)(memberOf=cn=service_mailman_access,ou=groups,dc=cccv,dc=de))', attributes=['uid'], search_scope=ldap3.LEVEL)
all_uids = [item['attributes']['uid'] for item in conn.response]
for uid in all_uids:
conn = get_conn()
conn.search(f'uid={uid},ou=users,dc=cccv,dc=de', '(objectClass=*)', attributes=['*'], search_scope=ldap3.BASE)
conn.compare('cn=service_mailman_access,ou=groups,dc=cccv,dc=de', 'uniqueMember', f'uid={uid},ou=users,dc=cccv,dc=de')
conn.compare('cn=cn=service_mailman_admin,ou=groups,dc=cccv,dc=de', 'uniqueMember', f'uid={uid},ou=users,dc=cccv,dc=de')
conn.compare('cn=cn=service_mailman_admin,ou=groups,dc=cccv,dc=de', 'uniqueMember', f'uid={uid},ou=users,dc=cccv,dc=de')
gc.collect()
list_filters = [
'(objectClass=posixAccount)',
'(objectClass=posixAccount)',
'(&(objectClass=posixAccount)(memberOf=cn=crew_crew,ou=groups,dc=cccv,dc=de))',
'(&(objectClass=posixAccount)(memberOf=cn=crew_crew_mod,ou=groups,dc=cccv,dc=de))',
# ... many more of these ...
]
conn = get_conn()
for list_filter in list_filters:
conn.search('ou=users,dc=cccv,dc=de', list_filter, attributes=['uid'])
```
Note how closing of connections is entirely handled by the garbage collector.
My guess is that our other workloads do a better job at closing connection and also don't open a new connection for every user. I.e. this workload here causes a lot more threads and I guess CacheControl is not entirely thread-safe.
A workaround for our problem is to disable caching. The fix is probably to get rid of CacheControl and implement caching on our own.https://git.cccv.de/uffd/uffd-ldapd/-/issues/3Improved caching2021-12-08T21:54:36ZJulianImproved cachingWe currently cache the http requests with CacheControl. If we get a search request for all users and afterwards a second one for a specific user, we perform two http api requests, although we could answer the second one with data we have...We currently cache the http requests with CacheControl. If we get a search request for all users and afterwards a second one for a specific user, we perform two http api requests, although we could answer the second one with data we have in cache from the first one.
A custom caching mechanism that works on the user/group data instead of http requests could do much better.https://git.cccv.de/uffd/uffd-ldapd/-/issues/2Connection timeouts2021-12-05T17:08:35ZJulianConnection timeoutsI've seen that e.g. wekan does not close LDAP connections at all if any exception occured. For such cases a timeout would be good.
However it is also quite common in LDAP applications to keep LDAP connections open for quite some time (h...I've seen that e.g. wekan does not close LDAP connections at all if any exception occured. For such cases a timeout would be good.
However it is also quite common in LDAP applications to keep LDAP connections open for quite some time (half an hour or so).