Release preparations

Added guard to first v2 migration in order to prevent accidental upgrades.
Extended the upgrade instructions and moved them from the README to a
standalone file.

README.md

@@ -41,13 +41,12 @@ export FLASK_APP=uffd
 flask group create 'uffd_access' --description 'Access to Single-Sign-On and Selfservice'
 flask group create 'uffd_admin' --description 'Admin access to uffd'
 flask role create 'base' --default --add-group 'uffd_access'
-flask role create 'admin' --default --add-group 'uffd_admin'
+flask role create 'admin' --add-group 'uffd_admin'
 flask user create 'testuser' --password 'userpassword' --mail 'test@example.com' --displayname 'Test User'
 flask user create 'testadmin' --password 'adminpassword' --mail 'admin@example.com' --displayname 'Test Admin' --add-role 'admin'
 ```
 
-Afterwards you can login as a normal user with "testuser" and "userpassword", or as an admin with "testad
-min" and "adminpassword".
+Afterwards you can login as a normal user with "testuser" and "userpassword", or as an admin with "testadmin" and "adminpassword".
 
 ## Deployment
 
@@ -72,43 +71,6 @@ The Debian package uses uwsgi to run uffd and ships an `uffd-admin` to execute f
 If you upgrade, make sure to run `flask db upgrade` after every update! The Debian package takes care of this by itself using uwsgi pre start hooks.
 For an example uwsgi config, see our [uswgi.ini](uwsgi.ini). You might find our [nginx include file](nginx.include.conf) helpful to setup a web server in front of uwsgi.
 
-## Migration from version 1
-
-If a custom config file name was set with `CONFIG_FILENAME`, this must be replaced with `CONFIG_PATH`.
-The new variable must be set to a full path instead of a filename relative to the application's instance directory.
-
-Prior to version 2 uffd stored users, groups and mail aliases in an LDAP server.
-To migrate from version 1 to a later version, make sure to keep the v1 config file as it is with all LDAP settings.
-Running the database migrations with `flask db upgrade` automatically imports all users, groups and mail forwardings from LDAP to the database.
-Note that all LDAP attributes must be readable, including the password field.
-Make sure to have a working backup of the database before running the database upgrade!
-Downgrading is not supported.
-
-After running the migrations you can remove all `LDAP_*`-prefixed settings from the config file except the following ones that are renamed:
-
-* `LDAP_USER_GID` -> `USER_GID`
-* `LDAP_USER_MIN_UID` -> `USER_MIN_UID`
-* `LDAP_USER_MAX_UID` -> `USER_MAX_UID`
-* `LDAP_USER_SERVICE_MIN_UID` -> `USER_SERVICE_MIN_UID`
-* `LDAP_USER_SERVICE_MAX_UID` -> `USER_SERVICE_MAX_UID`
-* `LDAP_GROUP_MIN_GID` -> `GROUP_MIN_GID`
-* `LDAP_GROUP_MAX_GID` -> `GROUP_MAX_GID`
-
-Upgrading will not perform any write access to the LDAP server.
-
-If the config option `ACL_SELFSERVICE_GROUP` is set but not `ACL_ACCESS_GROUP`, make sure to set `ACL_ACCESS_GROUP` to the same value as `ACL_SELFSERVICE_GROUP`,
-
-OAuth2 and API client definitions moved from the config (`OAUTH2_CLIENTS` and `API_CLIENTS_2`) to the database.
-The database migration automatically imports clients from the config.
-After upgrading the config options should be removed.
-
-Note that the `login_message` option is no longer supported for OAuth2 clients.
-The `required_group` is only correctly imported if it is set to a single group name (or absent).
-
-Also note that uffd can group OAuth2 and API clients of a service together.
-Set the `service_name` key in `OAUTH2_CLIENTS` and `API_CLIENTS_2` items to the same value to group them together.
-Without this key the import creates individual service objects for each client.
-
 ## Python Coding Style Conventions
 
 PEP 8 without double new lines, tabs instead of spaces and a max line length of 160 characters.

UPGRADE.md

+# Upgrading from v1 to v2
+
+Prior to v2 uffd stored users, groups and mail aliases on an LDAP server.
+OAuth2 and API client credentials were defined in the config. Starting with
+v2 uffd stores all of this in its database and no longer supports LDAP.
+A number of other features and configurations are no longer supported. See the
+changelog for details on changed and removed features.
+
+## Preparations
+
+Create a backup of the database before attempting an upgrade. The database
+migration scripts are quite complex and somewhat fragile. If anything fails,
+you will end up with a broken database that is difficult or impossible to
+recover from. Furthermore downgrading from v2 to v1 is not supported.
+
+Make sure no service (besides uffd) directly accesses your LDAP server.
+Migrate any remaining services to [uffd-ldapd][] or other solutions that
+solely rely on uffds API and OAuth2 endpoints. Uffd will cease to update
+any data stored in the LDAP directory.
+
+Migrate all API clients defined with the `API_CLIENTS` config option to the
+`API_CLIENTS_2` option. This includes changing authentication from a
+token-based mechanism to HTTP Basic Authentication and consequently replacing
+affected credentials.
+
+The imported OAuth2 and API clients are grouped by service objects. These
+service objects will be auto-created for each client with unique names derived
+from the `client_id` parameter. Add the `service_name` parameter to clients to
+set a custom name. This name is visible to users in place of the OAuth2
+`client_id`. Use the same `service_name` for multiple clients to group them
+together. This is recommended for OAuth2 and API credentials used by the same
+services, as future features like service-specific email addresses will be
+added as service-level options. The OAuth2 client parameter `required_group` is
+imported as a service-level option. Make sure that grouped OAuth2 clients have
+the same `required_group` value, otherwise nobody will be able to access the
+service. Note that values other than a single group name are not supported.
+
+Adjust the ACLs of your LDAP server so uffd can read the `userPassword`
+attribute of user objects. Note that uffd will not perform any writes to the
+LDAP server during or after the upgrade.
+
+If you use user bind (config option `LDAP_SERVICE_USER_BIND`), i.e. if you
+have uffd authenticate with the LDAP server using the credentials of the
+currently logged in user, you will have to replace this configuration and
+grant uffd full read access to all user, group and mail alias data with
+config-defined credentials.
+
+Install the new dependency `python3-argon2`. (Dist-)Upgrading the Debian
+package will do that for you. Do not uninstall the removed dependency
+`python3-ldap3` (i.e. do not run `apt autoremove`)! It is required to import
+data from the LDAP server.
+
+There is a safeguard in place to prevent accidental upgrades. Add the
+following line to your config file to disable the safeguard:
+
+```
+UPGRADE_V1_TO_V2=True
+```
+
+## Running the Upgrade
+
+Upgrade the Debian package to v2. This will restart the uffd UWSGI app. With
+the default UWSGI configuration, the database migration scripts will run
+automatically.
+
+Otherwise run them manually:
+
+```
+uffd-admin db upgrade
+```
+
+The database migration scripts import users, groups and mail aliases from the
+configured LDAP server. They also import OAuth2 and API clients defined with
+the `OAUTH2_CLIENTS` and `API_CLIENTS_2` config options to the database.
+
+Due to data being split between the LDAP server and the database, uffd v1
+tended to accumulate orphaned database objects (e.g. role memberships of
+deleted users). All orphaned objects are deleted during the upgrade.
+
+As a side-effect upgrading resets all rate limits.
+
+## Follow-up
+
+Rename the following config options:
+
+* `LDAP_USER_GID` -> `USER_GID`
+* `LDAP_USER_MIN_UID` -> `USER_MIN_UID`
+* `LDAP_USER_MAX_UID` -> `USER_MAX_UID`
+* `LDAP_USER_SERVICE_MIN_UID` -> `USER_SERVICE_MIN_UID`
+* `LDAP_USER_SERVICE_MAX_UID` -> `USER_SERVICE_MAX_UID`
+
+Add the following config options:
+
+* `GROUP_MIN_GID`
+* `GROUP_MAX_GID`
+
+Remove the following config options:
+
+* `UPGRADE_V1_TO_V2`
+* `LDAP_USER_SEARCH_BASE`
+* `LDAP_USER_SEARCH_FILTER`
+* `LDAP_USER_OBJECTCLASSES`
+* `LDAP_USER_DN_ATTRIBUTE`
+* `LDAP_USER_UID_ATTRIBUTE`
+* `LDAP_USER_UID_ALIASES`
+* `LDAP_USER_LOGINNAME_ATTRIBUTE`
+* `LDAP_USER_LOGINNAME_ALIASES`
+* `LDAP_USER_DISPLAYNAME_ATTRIBUTE`
+* `LDAP_USER_DISPLAYNAME_ALIASES`
+* `LDAP_USER_MAIL_ATTRIBUTE`
+* `LDAP_USER_MAIL_ALIASES`
+* `LDAP_USER_DEFAULT_ATTRIBUTES`
+* `LDAP_GROUP_SEARCH_BASE`
+* `LDAP_GROUP_SEARCH_FILTER`
+* `LDAP_GROUP_GID_ATTRIBUTE`
+* `LDAP_GROUP_NAME_ATTRIBUTE`
+* `LDAP_GROUP_DESCRIPTION_ATTRIBUTE`
+* `LDAP_GROUP_MEMBER_ATTRIBUTE`
+* `LDAP_MAIL_SEARCH_BASE`
+* `LDAP_MAIL_SEARCH_FILTER`
+* `LDAP_MAIL_OBJECTCLASSES`
+* `LDAP_MAIL_DN_ATTRIBUTE`
+* `LDAP_MAIL_UID_ATTRIBUTE`
+* `LDAP_MAIL_RECEIVERS_ATTRIBUTE`
+* `LDAP_MAIL_DESTINATIONS_ATTRIBUTE`
+* `LDAP_SERVICE_URL`
+* `LDAP_SERVICE_USE_STARTTLS`
+* `LDAP_SERVICE_BIND_DN`
+* `LDAP_SERVICE_BIND_PASSWORD`
+* `LDAP_SERVICE_USER_BIND`
+* `ENABLE_INVITE`
+* `ENABLE_PASSWORDRESET`
+* `ENABLE_ROLESELFSERVICE`
+* `OAUTH2_CLIENTS`
+* `API_CLIENTS` (should not be set, see "Preperation")
+* `API_CLIENTS_2`
+* `LDAP_SERVICE_MOCK` (development option, should not be set)
+
+If you set a custom config filename with the environment variable
+`CONFIG_FILENAME`, replace it with `CONFIG_PATH`. The new variable must be
+set to a full path instead of a filename.
+
+If you set the config option `ACL_SELFSERVICE_GROUP`, but not
+`ACL_ACCESS_GROUP`, make sure to set `ACL_ACCESS_GROUP` to the same value as
+`ACL_SELFSERVICE_GROUP`.
+
+Add a cron job that runs `uffd-admin cleanup` at least daily. Unless you
+modified `/etc/cron.d/uffd`, upgrading the Debian package will do this for you.
+
+Uninstall the previous dependency `python3-ldap3` (i.e. run `apt autoremove`).
+
+[uffd-ldapd]: https://git.cccv.de/uffd/uffd-ldapd

debian/install

@@ -2,3 +2,5 @@ uwsgi.ini /etc/uffd/
 nginx.include.conf /etc/uffd/
 debian/contrib/uffd.cfg /etc/uffd/
 debian/contrib/uffd-admin /usr/bin/
+README.md /usr/share/doc/uffd/
+UPGRADE.md /usr/share/doc/uffd/

uffd/migrations/versions/878b25c4fae7_ldap_to_db.py

@@ -23,6 +23,8 @@ def encode_filter(filter_params):
 	return '(&%s)'%(''.join(['(%s=%s)'%(attr, escape_filter_chars(value)) for attr, value in filter_params]))
 
 def get_ldap_conn():
+	if 'LDAP_SERVICE_URL' in current_app.config and not current_app.config.get('UPGRADE_V1_TO_V2'):
+		raise Exception('Refusing to run v1 to v2 migrations: UPGRADE_V1_TO_V2 not set. Make sure to read upgrade instructions first!')
 	critical = True
 	if 'LDAP_SERVICE_URL' not in current_app.config:
 		critical = False