eNBeWe authored 7 months ago and Julian committed 7 months ago

The return type of jwt.encode() changed from bytes in v1.x (Buster/Bullseye)
to str in v2.x (Bookworm). This let json.dumps crash on Buster und Bullseye
with "TypeError: Object of type bytes is not JSON serializable".

Flask v1.x (Buster/Bullseye) automatically uses simplejson.dumps instead of
json.dumps if it is installed. simplejson.dumps auto-converts bytes to str per
default. simplejson also happend to be installed in our CI images. This
prevented the bug from surfacing in CI tests. We removed simplejson from our
CI images in an external change.

Co-authored-by: Julian Rother <julian@cccv.de>

Prerequisite for implementing missing OIDC features.

Prerequisite for doing the same to OAuth2 state. This is required for
implementing missing OIDC features later.

Users with ACL_ACCESS_GROUP but without ACL_SELFSERVICE_GROUP were able to
access the 2FA setup pages. Like all selfservice pages, these pages should
only have been accessible to users with ACL_SELFSERVICE_GROUP.

Limited to OpenID provider conformance profiles "Basic" and "Config":

- Support for features mandatory to implement for all OpenID Providers,
  not the feature set for Dynamic OpenID Providers
- Only Authorization Code Flow, no support for Implicit/Hybrid Flow
- Only code response type, no support for token/id_token
- Server metadata is served at /.well-known/openid-configuration

Additional/optional features:

- Support for "claims" parameter
- Support for standard scopes "profile" and "email"
- Support for non-standard scope/claim "groups" (in violation of RFC 9068)

Compatability with existing (working) uffd client setups: Authorization
requests without the "openid" scope behave the same as before  Prior to this
change authorization requests with the "openid" scope were rejected by uffd.

This change adds direct dependencies to pyjwt and cryptography. Prior to this
change both were already transitive dependencies of oauthlib.

SQLAlchemy v1.4 (Debian Bookworm) annoyingly warns about select statements
that result in a cartesion product of multiple tables. We actually want
cartesion products in all affected cases, so we change "SELECT FROM a,b" to
the equivalent "SELECT FROM a JOIN b ON TRUE".

See https://docs.sqlalchemy.org/en/14/changelog/migration_14.html

Firefox autofills all type="password" inputs with passwords from its built-in
password store. This breaks usability of admin pages.

This change fixes that by adding autocomplete="new-password" to these inputs.
It also adds appropriate autocomplete attributes to other forms/inputs to
improve autocomplete behaviour across browsers:

- autocomplete="off" on all non-login/signup/selfservice forms
- autocomplete="new-password" or autocomplete="current-password" on all
  type="password" inputs to workaround Firefox's misdetection
- autocomplete="username"/"email"/"nickname" on login/signup/selfservice inputs
  wherever appropriate
- Avoid type="password" where possible (e.g. on readonly fields)

SQLAlchemy v1.4 (Debian Bookworm) annoyingly warns about overlapping
user/mfa_method relationships.

Fixes #146

Compatibility fix for Flask v2 (Debian Bookworm) and newer

byteplow authored 1 year ago and Julian committed 1 year ago

Automatically enabled based on OS/browser settings (prefers-color-scheme
CSS media query)

Co-authored-by: Julian Rother <julian@cccv.de>

9bfd6f81 changed the format of authorization codes, but did not adapt the
invalidation code accordingly. Because of this, authorization codes were
not invalidated and could have been used multiple times to request access
tokens until expiring.

Time-based one-time password (TOTP) codes are only valid for a short period
of time. In addition they are meant to be single-use to make them more
resistant against phishing and eavesdropping (e.g. keyloggers). Prior to this
change uffd did not keep track of used codes and thus did not prevent code
reuse.

- Fix apt package build on Bookworm
- Adapt babel.cfg to jinja 3.x.x and break compatability with older versions

- Add CI tests for Bookworm
- Disable pylint deprecation warnings for crypt
- Mitigate Flask changes that broke a few tests
- Set create_constraint=True for Booleans/Enums to mitigate SQLAlchemy changes
- Mitigate new Alembic CHECK constraint behaviour in batch mode

Recent setuptools releases refuse to build packages with invalid version
strings. So instead of using the bare commit hash as the version, we now
build proper version strings like X.Y.Z.dev-git.COMMIT for CI development
builds and X.Y.Z for release builds (same as before).

Alembic runs migration scripts on SQLite and MariaDB in auto-commit mode, so
inserting many rows with individual insert statements is extremely slow.

This setting is more flexible than the existing REMAILER_LIMIT_TO_USERS config
option. The config option is therefore deprecated and will be removed in the
next major version.

Turns check_migrations.py into a normal test case. Speeds up pipeline by
making html5validator use the artifacts from tests:buster:sqlite instead of
running the tests on its own.

Uffd now requires that MariaDB databases have utf8mb4 charset and
utf8mb4_nopad_bin collation. The collation was chosen for consistency with
SQLite's BINARY collation.

Previously Unix UIDs/GIDs were allocated by using the highest used ID + 1.
This caused ID reuse when the newest user/group was deleted. In addition, the
implementation did not work on MariaDB (at all, it was not possible to create
users/groups).

The new approach accounts for all IDs ever used regardless of whether or not
users/groups are deleted. It always allocates the lowest ID in the configured
range that was never used.

Aside from the different allocation algorithm, this change introduces a
generic locking mechanism and prerequisites for testing migration scripts.

Enforces uniqueness of (verified) email addresses across all users. Email
addresses are compared case-insensitivly and Unicode-normalized. The new
unique constraints are disabled by default and can be enabled with a CLI
command. They are planned to become mandatory in uffd v3.

A lot of software does not allow multiple users to share the same email
address. This change prevents problems with such software.

To enable this feature run the command:

  uffd-admin unique-email-addresses enable

The commands reports any issues (e.g. existing duplicate addresses) that
prevent enabling the feature.

This change also introduces a generic mechanism to store feature flags in the
database and improves error handling for login name constraint violations.

0bd26ee8 added __init__.py files to the tests subdirectory. This had two
unwanted side-effects:

1. setuptools.find_packages() recognised the tests as a package, so they were
   included in the pip and Debian packages.
2. The Debian package build process with dh_python automatically runs tests
   with unittest. Unittest's test discovery (in contrast to pytest) only works
   if __init__.py files exist, so this step did not do anything in the past.
   Now, failing tests caused the whole CI pipeline to fail very early without
   the helpful information provided by later stages.

This change disables running any tests during the Debian package build. It also
explicitly sets the package list to "uffd".

Restructure tests into views/models/commands subdirectories to mirror the new
source tree structure introduced with ac731bf4 (Restructure source tree).

Deprecates old case-sensitive format. Some software out there stores email
addresses converted to lower case, breaking v1 remailer addresses. The new
format is case-insensitive and generally more robust.

Uffd continues to use and support the v1 format for services setup before
this change. Support for the old format is planned to be remove in uffd v3.
It is possbile to gradually migrate services to the new format with a service
setting in the admin interface.

Also fixes compatability issue with very recent SQLAlchemy versions introduced
by b391e176 (whens parameter of case function).

Also fixes a minor email-related bug in the admin interface and bad
texts/translations in the selfservice UI.

Access control is done via normal api credentials.
See README.md for details.
Adds an optional dependency on python3-prometheus-client.

Preperation for future features that require per-service user settings
or state, e.g. stateful sync or service-specific email settings.

The additional JOIN of ServiceUser degrades getusers API performance
by 30-50%. For API calls that return many users, this is compensated by
an otherwise unrelated optimization (selectinload instead of joinedload).