merge ud-ldap with upstream

as discussed in the ldap docs, we have diverged from upstream userdir-ldap. remerge with the latest release, ideally reducing our diff to zero by contributing our patches back upstream.

this might imply some code changes to override some templates locally; through puppet instead of through the package, for example.

added Icebox label

mentioned in issue #34426 (closed)

changed the description

mentioned in issue #40363 (closed)

note that in #40182 (closed), i reduced that diff quite a bit, and submitted a pile of changes upstream. i haven't heard from DSA about those changes, unfortunately, but at least that part of the work was done. when I did those changes, i documented the new branching model and diff with upstream in the LDAP docs. to quote that document:

usedir-ldap-cgi fork status

In the last sync, usedir-ldap-cgi was brought from 27 patches down to 16, 10 of which were sent upstream. Our diff there is now:

  22 files changed, 11661 insertions(+), 553 deletions(-)

The large number of inserted lines is because we included the styleguide bootstrap.css which is 11561 lines on its own, so really, this is the diff stat if we ignore that stylesheet:

  21 files changed, 100 insertions(+), 553 deletions(-)

If the patches get merged upstream, our current delta is:

 21 files changed, 23 insertions(+), 527 deletions(-)

The only way forward here is either to make the "Debian" strings "variables" in the WML templates or completely remove the documentation from userdir-ldap-cgi (and move it to the project's respective wikis).

For now, we have changed the navigation to point to our wiki as much as possible. The next step is to remove our patches to the upstream documentation and make sure that documentation is not reachable to avoid confusion.

userdir-ldap fork status

Our diff in userdir-ldap is much smaller:

 6 files changed, 46 insertions(+), 19 deletions(-)

We have 4 patches there, and a handful were merged upstream. The remaining patches could probably live as configuration files in Puppet, reducing the diff to nil.

so the first target here should probably be to finish the latter, userdir-ldap. then the more difficult part is the -cgi stuff, where we should probably just strip the tricky docs out of the web interface, pointing to a (configurable) external document, since it's where most of the diff resides.

added LDAP label

mentioned in commit wiki-replica@b7541e34

i've attempted another merge, and the situation is not good.

to quote the wiki update i've just posted (wiki-replica@b7541e34):

2024 update

As of 2024-06-03, the situation has not improved:

anarcat@angela:userdir-ldap$ git diff dsa/master  --stat
 .gitlab-ci.yml               | 18 ------------------
 debian/changelog             | 22 ++++++++++++++++++++++
 debian/rules                 |  2 +-
 debian/ud-replicate.cron.d   |  2 +-
 misc/ud-update-sudopasswords |  4 ++--
 templates/passwd-changed     |  2 +-
 templates/welcome-message    | 41 ++++++++++++++++++++++++++++-------------
 test/test_pass.py            | 10 ++++++++++
 ud-mailgate                  | 14 ++++++++------
 ud-replicate                 |  4 ++--
 userdir-ldap.conf            |  2 +-
 userdir_ldap/generate.py     | 49 ++++++++++++++++++++++++++++++++++++++-----------
 12 files changed, 114 insertions(+), 56 deletions(-)

We seem incapable of getting our changes merged upstream at this point. Numerous patches were sent to DSA only to be either ignored, rewritten, or replaced without attribution. It has become such a problem that we have effectively given up on merging the two code bases.

We should acknowledge that some patches were actually merged, but the patches that weren't were so demotivating that it seems easier to just track this as a non-collaborating upstream, with our code as a friendly fork, than pretending there's real collaboration happening.

Our patch set is currently:

• tpo-scrub-0.3.104 (unchanged, possibly unmergeable):
  • 43c67a3 fix URL in passwd-changed template to torproject.org
  • f9f9a67 Set emailappend to torproject.org
  • c77a70b Use https:// in welcome email
  • 6966895 Re-apply tpo changes to Debian's repo
• mailpassword-generate-0.3.104 (patch rewritten upstream, unclear if still needed)
• hashpass-test-0.3.104 (unchanged)
  • 7ceb72b (add tests for ldap.HashPass, 2021-10-27 15:29:30 -0400)
• fix-crash-without-exim-0.3.104 (new)
  • 51716ed (ud-replicate: fix crash when exim is not installed, 2023-05-11 13:53:33 -0400)
• paramiko-workaround-0.3.104-dff949b (new, not sent upstream considering ssh-openssh-87 was rejected)
  • 6233f8e (workaround SSH host key lookup bug in paramiko, 2023-11-21 14:49:46 -0500)
• sshfp-openssh-87 (new, rejected)
  • 651f280 (disable SSHFP record for initramfs keys, 2023-05-10 14:38:56 -0400)
• py3_allowed_hosts_unicode-0.3.104) (new, rewritten upstream, conflicting)
  • 88bb60d (LDAP now returns bytes, fix another comparison in ud-mailgate, 2023-10-12 10:23:53 -0400)

The following patches were actually merged:

• bookworm-build-0.3.104:
  • d0740a9 (fix implicit int to str cast that broke in bookworm (bullseye?) upgrade, 2023-09-13)
  • 25d89bd fix warning about chown(1) call in bookworm
  • 9c49a4a fix Depends to support python3-only installs
  • 1ece069 bump dh compat to 7
  • 90ef120 make this build without python2
• install-restore-crash-0.3.104:
  • 4ab5d83 (fix crash: LDAP returns a string, cast it to an integer, 2023-09-14 10:28:41 -0400)
• procmail-0.3.104-pre-dd7f9a3:
  • 661875e (drop procmail from userdir-ldap dependencies, 2022-02-28 21:15:41 -0500)

Two patches are still in development:

• ssh-sk-0.3.104
  • a722f6f Add support for security key generated ssh public keys (sk- prefix).
• thunderbird-pgp
  • d026f0f (replace my email address with example one, 2024-06-03)
  • 8aca3fc (extract multipart mime message content correctly, 2024-06-03)
  • 90882c5 (fix sequoia signature parsing, 2024-06-03)
  • a395279 (fix TB signing test failure, 2024-06-03)
  • 618d55f (add testcase for thunderbird pgp mail, 2024-06-03)

Patches were not resent or rebased.

@weasel has shown interest in correcting this situation

i've actually done a merge of dsa/master in our master branch, but i'll deploy that in prod only tomorrow, and only after we merge the good work me and @lavamind did on fixing thunderbird support.

added Doing label and removed Icebox label

assigned to @anarcat

i've deployed the new merged version to production now and sent the tb/sequoia patch upstream. punting this back into the icebox until we hear from DSA.

added Icebox label and removed Doing label

also, @weasel feel free to take over this ticket!

unassigned @anarcat

mentioned in commit wiki-replica@fe225a88