This change is going to be backported to v1.x.x to have a good migration path. Bearer auth with API_CLIENTS config key is deprecated and planned to be removed in v2.0.0.
Pylint non-deterministicly fails to detect that a method is overwritten later and complains that the method does not exist. This is pretty annoying and remains unfixed in the most recent version.
CONFIG_FILENAME works relative to the app's instance path. While (strictly speaking) CONFIG_FILENAME is named correctly, it is not really obvious that it should be set to an instance-relative filename instead of a path. The current uwsgi.ini file illustrates this problem. Not having a way to specify an absolute config file path is a problem for the Debian package: The actual config file /etc/uffd/uffd.cfg must be symlinked to /usr/share/uffd/instance/config.cfg to be found. Setting CONFIG_PATH to "/etc/uffd/uffd.cfg" simplifies this. Since this change is part of a new major release, we can drop CONFIG_FILENAME in favour of CONFIG_PATH.
Previously ACL_ACCESS_GROUP defaulted to the value of ACL_SELFSERVICE_GROUP, now it defaults to "uffd_access". Note that ACL_SELFSERVICE_GROUP has the same default value. If you set ACL_SELFSERVICE_GROUP to a different value but not ACL_ACCESS_GROUP, you will need to update your config.
The options were introduced to cleanly handle LDAP user connections. Since LDAP support is now gone and hence user connections are gone too, these options are no longer necessary. While the options may be useful in other cases, we cannot continuously test them and so we are removing them for now.
Previously the getmails API endpoint did not match "receive_address" values case-insensitivly like it did pre-v2. To solve this independent of database collations, all existing mail receive addresses are converted to lower-case and new/changed receive addresses are constraint to ASCII lower-case letters, digits and symbols.
Previously User used salted SHA512 with OpenLDAP-style prefix syntax and Signup used crypt. Both models had their own hashing and verification code. Now both use OpenLDAP-style syntax with support for all traditional formats including crypt. Salted SHA512 is used for new User and Signup passwords. Existing Signup objects are migrated to the new format and remain functional. User passwords now support gradual migration to another hash algorithm when it is changed in the future. This code is planned to be used for database-stored API and OAuth2 client secrets.
Argon2 is a modern password hashing algorithm. It is significantly more secure than the previous algorithm (salted SHA512). User logins with Argon2 are relativly slow and cause significant spikes in CPU and memory (100MB) usage. Existing passwords are gradually migrated to Argon2 on login.
Calling op.get_bind outside a callback broke "flask db history".
The command replaces all existing mechanisms for deleting expired objects. It should run at least daily. The Debian package includes a corresponding cron job. Ratelimit events now use UTC timestamps instead of localtime. On upgrade all past ratelimit events are cleared.
The generation now happens in a subquery inside the INSERT statement instead of separate client-managed query. This should also reduce the risk of race conditions. Service and non-service users may now use the same UID range.
Also adds a shallow Service model that coexists with the config-defined services to group multiple OAuth2 and API clients together. Clients defined in the config with OAUTH2_CLIENTS and API_CLIENTS_2 are imported by the database migrations. Removes support for complex values for the OAuth2 client group_required option. Only simple group names are supported, not (nested) lists of groups previously interpreted as AND/OR conjunctions. Also removes support for the login_message parameter of OAuth2 clients.
The migration originally failed to convert the passwords/secrets to the format expected by PasswordHash resulting in invalid password hashes. With this change, the migration works correctly. Also fixes minor template bug.
The original change completely broke single logout support. The migration now uses the correct hashing algorithm (unsalted SHA512 instead of salted SHA512) for OAuth2/API secrets/passwords.
Added guard to first v2 migration in order to prevent accidental upgrades. Extended the upgrade instructions and moved them from the README to a standalone file.
When the service overview was introduced, it was meant to be optional. Thus if the SERVICES config option was empty (the default), uffd returned 404. Commit fa67bde0 (Migrate OAuth2 and API clients to database) introduced the regression that accessing the service overview page when no services are visible based on the permissions of the current user (or guest if not logged in), 404 is returned. This change fixes the regression and further changes the behavior to improve consistency. Since fa67bde0, the page is relevant to admin users regardless of the SERVICES config option. Therefore uffd asks for login or reports missing permissions in all cases it originally returned 404.
Closes #152 Co-authored-by: davidc
The group and role update subcommands set the description to an empty string if the "--description" option was ommitted. Fixes #156
When the "new invite" page was submitted with e.g. an invalid "Valid Until" value, uffd displayed an error and reset the whole form. This was confusing to users. Now the form content is preserved on errors. Also the "Valid Until" field now has min/max attributes to prevent submitting the form with invalid values. Fixes #134
With this feature, uffd can be configured to hide mail addresses of users from certain services while still allowing the services to send mails to the users. To these services uffd returns special remailer addresses instead of the real mail addresses. When a service sends an email to a remailer address the mail server queries uffd's API and replaces the remailer address with the real mail address in both envelope and headers. This feature requires additional mail server configuration (Postfix canonical_maps) and support in uffd-socketmapd.
Move all models, views, cli commands and templates into corresponding top-level folders. Detailed changes: - uffd/<NAME>/models.py -> uffd/models/<NAME>.py - uffd/<NAME>/cli.py -> uffd/commands/<NAME>.py - uffd/<NAME>/views.py -> uffd/views/<NAME>.py - uffd/<NAME>/templates/* -> uffd/templates/ - uffd/ratelimit.py -> uffd/models/ratelimit.py (it contains models) - gendevcert from uffd/__init__.py -> uffd/commands/gendevcert.py - profile from uffd/__init__.py -> uffd/commands/profile.py - cleanup from uffd/tasks.py -> uffd/commands/cleanup.py - roles-update-all from uffd/role/views.py -> uffd/commands/... - Views from uffd/__init__.py -> uffd/views/__init__.py - All models can/should be imported from uffd.models - flask shell auto-imports all models instead of only a few The old structure was meant to keep the code modular and related code/resources close to each other. However, the modules turned out to be heavily interdependent and not very modular. Also importing was fragile due to ordering issues. With the new structure the dependency tree is much simpler: Infrastructure code (top-level *.py files) has no internal dependencies. Models only depend on infrastructure and other models. Views and cli commands depend on infrastructure, models and other views/commands. Going forward there is still some restructuring to do, e.g.: - Move mfa setup views to selfservice views - Move mfa auth views to session views - Move utility code from views to infrastructure (e.g. login_required) - In most cases views should not need to import from other views - Reorganize infrastructure code
Unittest jobs now fail if any test fails. Unittests on Bullseye no longer fail due to jinja2 import errors. Linter jobs run faster.
Convert DateTime fields to UTC, use "utcnow" instead of "now" and use babel helper/filter when dates/times are displayed or parsed from user input. Uffd continues to use the system's timezone in the user interface by default. However, it is now possible to overwrite this with the BABEL_DEFAULT_TIMEZONE config option.
Preperation for future features that require per-service user settings or state, e.g. stateful sync or service-specific email settings. The additional JOIN of ServiceUser degrades getusers API performance by 30-50%. For API calls that return many users, this is compensated by an otherwise unrelated optimization (selectinload instead of joinedload).
Access control is done via normal api credentials. See README.md for details. Adds an optional dependency on python3-prometheus-client.
Also fixes a minor email-related bug in the admin interface and bad texts/translations in the selfservice UI.
Deprecates old case-sensitive format. Some software out there stores email addresses converted to lower case, breaking v1 remailer addresses. The new format is case-insensitive and generally more robust. Uffd continues to use and support the v1 format for services setup before this change. Support for the old format is planned to be remove in uffd v3. It is possbile to gradually migrate services to the new format with a service setting in the admin interface. Also fixes compatability issue with very recent SQLAlchemy versions introduced by b391e176 (whens parameter of case function).
0bd26ee8 added __init__.py files to the tests subdirectory. This had two unwanted side-effects: 1. setuptools.find_packages() recognised the tests as a package, so they were included in the pip and Debian packages. 2. The Debian package build process with dh_python automatically runs tests with unittest. Unittest's test discovery (in contrast to pytest) only works if __init__.py files exist, so this step did not do anything in the past. Now, failing tests caused the whole CI pipeline to fail very early without the helpful information provided by later stages. This change disables running any tests during the Debian package build. It also explicitly sets the package list to "uffd".
Enforces uniqueness of (verified) email addresses across all users. Email addresses are compared case-insensitivly and Unicode-normalized. The new unique constraints are disabled by default and can be enabled with a CLI command. They are planned to become mandatory in uffd v3. A lot of software does not allow multiple users to share the same email address. This change prevents problems with such software. To enable this feature run the command: uffd-admin unique-email-addresses enable The commands reports any issues (e.g. existing duplicate addresses) that prevent enabling the feature. This change also introduces a generic mechanism to store feature flags in the database and improves error handling for login name constraint violations.
Previously Unix UIDs/GIDs were allocated by using the highest used ID + 1. This caused ID reuse when the newest user/group was deleted. In addition, the implementation did not work on MariaDB (at all, it was not possible to create users/groups). The new approach accounts for all IDs ever used regardless of whether or not users/groups are deleted. It always allocates the lowest ID in the configured range that was never used. Aside from the different allocation algorithm, this change introduces a generic locking mechanism and prerequisites for testing migration scripts.
Uffd now requires that MariaDB databases have utf8mb4 charset and utf8mb4_nopad_bin collation. The collation was chosen for consistency with SQLite's BINARY collation.
Turns check_migrations.py into a normal test case. Speeds up pipeline by making html5validator use the artifacts from tests:buster:sqlite instead of running the tests on its own.
This setting is more flexible than the existing REMAILER_LIMIT_TO_USERS config option. The config option is therefore deprecated and will be removed in the next major version.
Alembic runs migration scripts on SQLite and MariaDB in auto-commit mode, so inserting many rows with individual insert statements is extremely slow.
Recent setuptools releases refuse to build packages with invalid version strings. So instead of using the bare commit hash as the version, we now build proper version strings like X.Y.Z.dev-git.COMMIT for CI development builds and X.Y.Z for release builds (same as before).
- Add CI tests for Bookworm - Disable pylint deprecation warnings for crypt - Mitigate Flask changes that broke a few tests - Set create_constraint=True for Booleans/Enums to mitigate SQLAlchemy changes - Mitigate new Alembic CHECK constraint behaviour in batch mode
- Fix apt package build on Bookworm - Adapt babel.cfg to jinja 3.x.x and break compatability with older versions
Time-based one-time password (TOTP) codes are only valid for a short period of time. In addition they are meant to be single-use to make them more resistant against phishing and eavesdropping (e.g. keyloggers). Prior to this change uffd did not keep track of used codes and thus did not prevent code reuse.
Automatically enabled based on OS/browser settings (prefers-color-scheme
CSS media query)
Co-authored-by: 
Julian Rother <julian@cccv.de>
Compatibility fix for Flask v2 (Debian Bookworm) and newer
Firefox autofills all type="password" inputs with passwords from its built-in password store. This breaks usability of admin pages. This change fixes that by adding autocomplete="new-password" to these inputs. It also adds appropriate autocomplete attributes to other forms/inputs to improve autocomplete behaviour across browsers: - autocomplete="off" on all non-login/signup/selfservice forms - autocomplete="new-password" or autocomplete="current-password" on all type="password" inputs to workaround Firefox's misdetection - autocomplete="username"/"email"/"nickname" on login/signup/selfservice inputs wherever appropriate - Avoid type="password" where possible (e.g. on readonly fields)
SQLAlchemy v1.4 (Debian Bookworm) annoyingly warns about select statements that result in a cartesion product of multiple tables. We actually want cartesion products in all affected cases, so we change "SELECT FROM a,b" to the equivalent "SELECT FROM a JOIN b ON TRUE". See https://docs.sqlalchemy.org/en/14/changelog/migration_14.html
Limited to OpenID provider conformance profiles "Basic" and "Config": - Support for features mandatory to implement for all OpenID Providers, not the feature set for Dynamic OpenID Providers - Only Authorization Code Flow, no support for Implicit/Hybrid Flow - Only code response type, no support for token/id_token - Server metadata is served at /.well-known/openid-configuration Additional/optional features: - Support for "claims" parameter - Support for standard scopes "profile" and "email" - Support for non-standard scope/claim "groups" (in violation of RFC 9068) Compatability with existing (working) uffd client setups: Authorization requests without the "openid" scope behave the same as before Prior to this change authorization requests with the "openid" scope were rejected by uffd. This change adds direct dependencies to pyjwt and cryptography. Prior to this change both were already transitive dependencies of oauthlib.
Users with ACL_ACCESS_GROUP but without ACL_SELFSERVICE_GROUP were able to access the 2FA setup pages. Like all selfservice pages, these pages should only have been accessible to users with ACL_SELFSERVICE_GROUP.
Prerequisite for doing the same to OAuth2 state. This is required for implementing missing OIDC features later.
Prerequisite for implementing missing OIDC features.
The return type of jwt.encode() changed from bytes in v1.x (Buster/Bullseye)
to str in v2.x (Bookworm). This let json.dumps crash on Buster und Bullseye
with "TypeError: Object of type bytes is not JSON serializable".
Flask v1.x (Buster/Bullseye) automatically uses simplejson.dumps instead of
json.dumps if it is installed. simplejson.dumps auto-converts bytes to str per
default. simplejson also happend to be installed in our CI images. This
prevented the bug from surfacing in CI tests. We removed simplejson from our
CI images in an external change.
Co-authored-by: Julian Rother <julian@cccv.de>