ldap.md

LDAP is a directory service we use to inventory the users, groups, passwords, (some) email forwards and machines.

• Tutorial
• How-to
  • Connecting to LDAP
  • Restoring from backups
    • Partial restore
    • Full rollback
  • Listing members of a group
  • Searching LDAP
  • Modifying the schema
  • Pager playbook
    • LDAP server failure
    • Git server failure
    • Deadlocks in ud-replicate
    • Troubleshooting changes@ failures
    • Dependency loop on new installs
  • Disaster recovery
• Reference
  • Installation
  • SLA
  • Design
    • Architecture overview
    • Configuration file creation and distribution
    • LDAP user fields
      • sudoPassword field format
      • sshRSAAuthKey field format
    • LDAP host fields
      • exportOptions values
    • Files managed by ud-generate
    • Email gateway
    • Interactions with Puppet
    • DNS zone file management
    • Source file analysis
  • Issues
  • Monitoring and testing
  • Logs and metrics
  • Backups
  • Other documentation
• Discussion
  • Overview
  • Goals
    • Must have
    • Nice to have
    • Non-Goals
  • Approvals required
  • Proposed Solution
  • Cost
  • Alternatives considered

Tutorial

The main LDAP documentation is on db.torproject.org. See specifically the instructions on how to:

• reset a lost password
• change your forwarding address
• change your SSH key

The rest of this document is targeted at sysadmins troubleshooting LDAP issues, setting up new services, or trying to understand the setup.

How-to

Connecting to LDAP

LDAP is not accessible to the outside world, so you need to get behind the firewall. Most operations are done directly on the LDAP server, by logging in as a regular user on db.torproject.org (currently alberti).

Once that's resolved, you can use ldapvi(1) or ldapsearch(1) to inspect the database. User documentation on that process is in doc/accounts and https://db.torproject.org. See also the rest of this documentation.

Restoring from backups

There's no special backup procedures for the LDAP server: it's backed up like everything else in the howto/backup system.

To restore the OpenLDAP database, you need to head over the Bacula director, and enter the console:

ssh -tt bacula-director-01 bconsole

Then call the restore command and select 6: Select backup for a client before a specified time. Then pick the server (currently alberti.torproject.org) and a date. Then you need to "mark" the right files:

cd /var/lib/ldap
mark *
done

Then confirm the restore. The files will end up in /var/tmp/bacula-restores on the LDAP server.

The next step depends on whether this is a partial or total restore.

Partial restore

If you only need to access a specific field or user or part of the database, you can use slapcat to dump the database from the restored files even if the server is not running. You first need to "configure" a "fake" server in the restore directory. You will need to create two files under /var/tmp/bacula-restores:

• /var/tmp/bacula-restores/etc/ldap/slapd.conf
• /var/tmp/bacula-restores/etc/ldap/userdir-ldap-slapd.conf

They can be copied from /etc, with the following modificatiosn:

diff -ru /etc/ldap/slapd.conf etc/ldap/slapd.conf
--- /etc/ldap/slapd.conf	2011-10-30 15:43:43.000000000 +0000
+++ etc/ldap/slapd.conf	2019-11-25 19:48:57.106055596 +0000
@@ -17,10 +17,10 @@
 
 # Where the pid file is put. The init.d script
 # will not stop the server if you change this.
-pidfile         /var/run/slapd/slapd.pid
+pidfile         /var/tmp/bacula-restores/var/run/slapd/slapd.pid
 
 # List of arguments that were passed to the server
-argsfile        /var/run/slapd/slapd.args
+argsfile        /var/tmp/bacula-restores/var/run/slapd/slapd.args
 
 # Read slapd.conf(5) for possible values
 loglevel        none
@@ -57,4 +57,4 @@
 #backend		<other>
 
 # userdir-ldap
-include /etc/ldap/userdir-ldap-slapd.conf
+include /var/tmp/bacula-restores/etc/ldap/userdir-ldap-slapd.conf
diff -ru /etc/ldap/userdir-ldap-slapd.conf etc/ldap/userdir-ldap-slapd.conf
--- /etc/ldap/userdir-ldap-slapd.conf	2019-11-13 20:55:58.789411014 +0000
+++ etc/ldap/userdir-ldap-slapd.conf	2019-11-25 19:49:45.154197081 +0000
@@ -5,7 +5,7 @@
 suffix          "dc=torproject,dc=org"
 
 # Where the database file are physically stored
-directory       "/var/lib/ldap"
+directory       "/var/tmp/bacula-restores/var/lib/ldap"
 
 moduleload      accesslog
 overlay accesslog
@@ -123,7 +123,7 @@
 
 
 database hdb
-directory       "/var/lib/ldap-log"
+directory       "/var/tmp/bacula-restores/var/lib/ldap-log"
 suffix cn=log
 #
 sizelimit 10000

Then slapcat is able to read those files directly:

slapcat -f /var/tmp/bacula-restores/etc/ldap/slapd.conf -F /var/tmp/bacula-restores/etc/ldap

Copy-paste the stuff you need into ldapvi.

Full rollback

Untested procedure.

If you need to roll back the entire server to this version, you first need to stop the LDAP server:

service slapd stop

Then move the files into place (in /var/lib/ldap):

mv /var/lib/ldap{,.orig}
cp -R /var/tmp/bacula-restores/var/lib/ldap /var/lib/ldap
chown -R openldap:openldap /var/lib/ldap

And start the server again:

service slapd start

Listing members of a group

To tell which users are part of a given group (LDAP or otherwise), you can use the getent(1) command. For example, to see which users are part of the tordnsel group, you would call this command:

$ getent group tordnsel
tordnsel:x:1532:arlo,arma

In the above, arlo and arma are members of the tordnsel group. The fields in the output are in the format of the group(5) file.

Note that the group membership will vary according to the machie on which the command is run, as not all users are present everywhere.

Searching LDAP

This will load a text editor with a dump of all the users (useful to modify an existing user or add a new one):

ldapvi -ZZ --encoding=ASCII --ldap-conf -h db.torproject.org -D "uid=$USER,ou=users,dc=torproject,dc=org"

This will list all known hosts in LDAP:

ldapsearch -ZZ -vLxW -h db.torproject.org -D "uid=$USER,ou=users,dc=torproject,dc=org" -b "ou=hosts,dc=torproject,dc=org" '(objectclass=*)' | grep ^dn:

Note that this will only work on the LDAP host itself or on whitelisted hosts which are few right now. This is mostly documented for TPA members.

Modifying the schema

If you need to add, change or remove a field in the schema of the LDAP database, it is a different, and complex operation. You will only need to do this if you launch a new service that (say) requires a new password specifically for that service.

The schema is maintained in the userdir-ldap.git repository. It is stored in the userdir-ldap.schema file. Assuming the modified object is a user, you would need to edit the file in three places:

1. as a comment, in the beginning, to allocate a new field, for example:

   @@ -113,6 +113,7 @@
    #   .45 - rebootPolicy
    #   .46 - totpSeed
    #   .47 - sshfpHostname
   +#   .48 - mailPassword
    #
    # .3 - experimental LDAP objectClasses
    #   .1 - debianDeveloper

This is purely informative, but it is important as it serves as a central allocation point for that numbering system. Also note that the entire schema lives under a branch of the Debian.org IANA OID allocation.

2. create the actual attribute, somewhere next to a similar attribute or after the previous OID, in this case we created an attributed called mailPassword right after rtcPassword, since other passwords were also grouped there:

   attributetype ( 1.3.6.1.4.1.9586.100.4.2.48
          NAME 'mailPassword'
          DESC 'mail password for SMTP'
          EQUALITY octetStringMatch
          SYNTAX 1.3.6.1.4.1.1466.115.121.1.40 )

3. finally, the new attribute needs to be added to the objectclass. in our example, the field was added alongside the other password fields in the debianAccount objectclass, which looked like this after the change:

   objectclass ( 1.3.6.1.4.1.9586.100.4.1.1
   	NAME 'debianAccount'
   	DESC 'Abstraction of an account with POSIX attributes and UTF8 support'
   	SUP top AUXILIARY
   	MUST ( cn $ uid $ uidNumber $ gidNumber )
   	MAY ( userPassword $ loginShell $ gecos $ homeDirectory $ description $ mailDisableMessage $ sudoPassword $ webPassword $ rtcPassword $ mailPassword $ totpSeed ) )

Once that schema file is propagated to the LDAP server, this should automatically be loaded by slapd when it is restarted (see below). But the ACL for that field should also be modified. In our case, we had to add the mailPassword field to two ACLs:

--- a/userdir-ldap-slapd.conf.in
+++ b/userdir-ldap-slapd.conf.in
@@ -54,7 +54,7 @@ access to attrs=privateSub
        by * break
 
 # allow users write access to an explicit subset of their fields
-access to attrs=c,l,loginShell,ircNick,labeledURI,icqUIN,jabberJID,onVacation,birthDate,mailDisableMessage,gender,emailforward,mailCallout,mailGreylisting,mailRBL,mailRHSBL,mailWhitelist,mailContentInspectionAction,mailDefaultOptions,facsimileTelephoneNumber,telephoneNumber,postalAddress,postalCode,loginShell,onVacation,latitude,longitude,VoIP,userPassword,sudoPassword,webPassword,rtcPassword,bATVToken
+access to attrs=c,l,loginShell,ircNick,labeledURI,icqUIN,jabberJID,onVacation,birthDate,mailDisableMessage,gender,emailforward,mailCallout,mailGreylisting,mailRBL,mailRHSBL,mailWhitelist,mailContentInspectionAction,mailDefaultOptions,facsimileTelephoneNumber,telephoneNumber,postalAddress,postalCode,loginShell,onVacation,latitude,longitude,VoIP,userPassword,sudoPassword,webPassword,rtcPassword,mailPassword,bATVToken
        by self write
        by * break
 
@@ -64,7 +64,7 @@ access to attrs=c,l,loginShell,ircNick,labeledURI,icqUIN,jabberJID,onVacation,bi
 ##
 
 # allow authn/z by anyone
-access to attrs=userPassword,sudoPassword,webPassword,rtcPassword,bATVToken
+access to attrs=userPassword,sudoPassword,webPassword,rtcPassword,mailPassword,bATVToken
        by * compare
 
 # readable only by self

If those are the only required changes, it is acceptable to directly make those changes directly on the LDAP server, as long as the exact same changes are performed in the git repositoy.

It is preferable, however, to build and upload userdir-ldap as a Debian package instead.

Pager playbook

An LDAP server failure can trigger lots of emails as ud-ldap fails to synchronize things. But the infrastructure should survive the downtime, because users and passwords are copied over to all hosts. In other words, authentication doesn't rely on the LDAP server being up.

In general, OpenLDAP is very stable and doesn't generally crash, so we haven't had many emergencies scenarios with it yet. If anything happens, make sure the slapd service is running.

The ud-ldap software, on the other hand, is a little more complicated and can be hard to diagnose. It has a large number of moving parts (Python, Perl, Bash, Shell scripts) and talks over a large number of protocols (email, DNS, HTTPS, SSH, finger). The failure modes documented here are far from exhaustive and you should expect exotic failures and error messages.

LDAP server failure

That said, if the LDAP server goes down, password changes will not work, and the server inventory (at https://db.torproject.org/) will be gone. A mitigation is to use Puppet manifests and/or PuppetDB to get a host list and server inventory, see the Puppet documentation for details.

Git server failure

The LDAP server will fail to regenerate (and therefore update) zone files and zone records if the Git server is unavailable. This is described in issue 33766. The fix is to recover the git server. A workaround is to run this command on the primary DNS server (currently nevii):

sudo -u dnsadm /srv/dns.torproject.org/bin/update --force

Deadlocks in ud-replicate

The ud-replicate process keeps a "reader" lock on the LDAP server. If for some reason the network transport fails, that lock might be held on forever. This happened in the past on hosts with flaky network or ipsec problems that null-routed packets between ipsec nodes.

There is a Nagios check that will detect stale synchronisations. Example:

Subject: ** PROBLEM Service Alert: palmeri/setup - ud-ldap freshness is WARNING **

Note that this can generate a lot of warnings because one per server will be sent!

The fix is to find the offending locked process and kill it. In desperation:

pkill -u sshdist rsync

... but really, you should carefully review the rsync processes before killing them all like that. And obviously, fixing the underlying network issue would be important to avoid such problems in the future.

Also note that the lock file is in /var/cache/userdir-ldap/hosts/ud-generate.lock, and ud-generate tries to get a write lock on the file. This implies that a deadlock will also affect file generation and keep ud-generate from generating fresh config files.

Finally, ud-replicate also holds a lock on /var/lib/misc on the client side, but that rarely causes problems.

Troubleshooting changes@ failures

A common user question is that they are unable to change their SSH key. This can happen if their email client somehow has trouble sending a PGP signature correctly. Most often than not, this is because their email client does a line wrap or somehow corrupts the OpenPGP signature in the email.

A good place to start looking for such problems is the log files on the LDAP server (currently alberti). For example, this has a trace of all the emails received by the changes@ alias:

/srv/db.torproject.org/mail-logs/received.changes

A common problem is people using --clearsign instead of --sign when sending an SSH key. When that hapepns, many email clients (including Gmail) will word-wrap the SSH key after the comment, breaking the signature. For example, this might happen:

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKxqYYEeus8dRXBHhLsp0SjH7ut2X8UM9hdXN=
wJIl89otcJ5qKoXj90K9hq8eBjG2KuAZtp0taGQHqzBOFK+sFm9/gIqvzzQ07Pn0xtkmg10Hunq=
vPKMj4gDFLIqTF0WSPA2E6L/TWaeVJ+IiGuE49j+0Ohd7UFDEquM1H/zno22vIEm/dxWLPWD9gG=
MmwBghvfK/dRyzSEDGlAVeWLzoIvVOG12/ANgic3TlftbhiLKTs52hy8Qhq/aQBqd0McaE4JGxe=
9k71OCg+0WHVS4q7HVdTUqT3VFFfz0kjDzYTYQQcHMqPHvYzZghxMVCmteNdJNwJmGSNPVaUeJG=
MumJ9
anarcat@curie

-----BEGIN PGP SIGNATURE-----
[...]
-----END PGP SIGNATURE-----

Using --sign --armor will work around this problem, as the original message will all be ascii-armored.

Dependency loop on new installs

Installing a new server requires granting the new server access various machines, including puppet and the LDAP server itself. This is granted ... by Puppet through LDAP!

So a server cannot register itself on the LDAP server and needs an operator to first create a host snippet on the LDAP server, and then run Puppet on the Puppet server. This is documented in the installation notes.

Disaster recovery

The LDAP server is mostly built by hand and should therefore be restored from backups in case of a catastrophic failure. Care should be taken to keep the SSH keys of the server intact.

The IP address (and name?) of the LDAP server should not be hardcoded anywhere. When the server was last renumbered (issue 33908), the only changes necessary were on the server itself, in /etc. So in theory, a fresh new server could be deployed (from backups) in a new location (and new address) without having to do much.

Reference

Installation

TODO: explain how an LDAP server could be installed from scratch.

SLA

The LDAP server is designed to be fault-tolerant in the sense that it's database is copied over other hosts. It should otherwise be highly available as it's a key component in managing users authentication and authorization, and machines.

Design

The LDAP setup at Tor is based on the one from Debian.org. /etc/password and groups files are synchronized from the central LDAP server using the sshdist account, which means things keep working when LDAP is down. Most operations can be performed on the db.torproject.org site or by email.

Architecture overview

This is all implemented by a tool called ud-ldap, inherited from the Debian project. The project is made of a collection of bash, Python and Perl scripts which take care of synchronizing various configuration files to hosts based on the LDAP configuration. Most of this section aims at documenting how this program works.

ud-ldap is made of two Debian packages: userdir-ldap, which ships the various server- and client-side scripts (and is therefore installed everywhere), and userdir-ldap-cgi which ships the web interface (and is therefore installed only on the LDAP server).

Configuration files are generated on the server by the ud-generate command, which goes over the LDAP directory and crafts a tree of configuration files, one directory per host defined in LDAP. Then each host pulls those configuration files with ud-replicate. A common set of files is exported everywhere, while the exportOptions field can override that by disabling some exports or enabling special ones.

An email gateway processes OpenPGP-signed emails which can change a user's fields, passwords or SSH keys, for example.

TODO: LDAP diagram missing ud-mailgate and eugeni

Configuration file creation and distribution

An important part of ud-ldap is the ud-generate command, which generates configuration files for each host. Then the ud-replicate command runs on each node to rsync those files. Both commands are ran from cron on regular intervals, the latter defined in Puppet, the former hardcoded to 15 minutes.

More specifically, this is what happens:

1. on the LDAP server (currently alberti), ud-generate writes various files to /var/cache/userdir-ldap/hosts/, one directory per host

2. on all hosts, ud-replicate rsync's that host's directory from the LDAP server (as the sshdist user) to /var/lib/misc/$HOSTNAME and a symlink ensures /var/lib/misc/thishost points to that directory

TODO: walk through ud-generate. more explicitely.

LDAP user fields

User field	Meaning
cn	"common name" AKA "last name"
emailForward	address to forward email to
gecos	GECOS metadata field
gidNumber	Primary numeric group identifier, the UNIX GID
homeDirectory	UNIX $HOME location, unused
ircNick	IRC nickname, informative
keyFingerprint	OpenPGP fingerprint, grants access to email gateway
labeledURI	home page?
loginShell	UNIX login shell, grants user shell access, depending on gidNumber
mailCallout	?
mailContentInspectionAction	?
mailDefaultOptions	?
mailGreylisting	?
mailDisableMessage	message to bounce messages with to disable an email account
rtcPassword	previously used in XMPP authentication, unused
samba*	many samba fields, unused
shadowExpire	Account expiry (in days?)
shadowInactive	?
shadowLastChange	Last change date (epoch seconds?)
shadowMax	?
shadowMin	?
shadowWarning	?
sn	"surname" AKA "first name"
sshRSAAuthKey	SSH public keys
sudoPassword	sudo passwords on different hosts
supplementaryGid	Extra groups GIDs the user is a member of
uidNumber	Numeric user identifier, the UNIX UID, not to be confused with the above
uid	User identifier, the user's name
userPassword	LDAP password field, stripped of the {CRYPT} prefix to be turned into a UNIX password if relevant

sudoPassword field format

The sudoPassword field is special. It has 4 fields seperated by spaces:

1. a UUID
2. the status, which is either the string unconfirmed or the string confirmed: followed by a SHA1 (!) HMAC of the string password-is-confirmed, sudo, the UID, the UUID, the host list, and the hashed password, joined by colons (:), primed with a secret key stored in /etc/userdir-ldap/key-hmac-$UID where UID is the numeric identifier of the calling user, generally 33 (probably the webserver?) or sshdist? The secret key can also overridden by the UD_HMAC_KEY environment variable
3. the host list, either * (meaning all hosts) or a comma (,) separated list of hosts this password applies to
4. the hashed password, which is restricted to 50 characters: if longer, it is invalid (*)

That password field gets validated by email through ud-mailgate.

The field can, of course, have multiple values.

sshRSAAuthKey field format

The sshRSAAuthKey field can have multiple values. Each one should be a valid authorized_keys(5) file.

Its presence influences whether a user is allowed to login to a host or not. That is, if it is missing, the user will not be added to the shadow database.

The GITOLITE hosts treat the field specially: it looks for allowed_hosts fields and will match only on the right host. If will skip keys that have other options.

LDAP host fields

Group field	Meaning
description	free-form text field description
memory	main memory size, with M suffix (unused?)
disk	main disk size, with G suffixed (unused?)
purpose	like description but purpose of the host
architecture	CPU architecture (e.g. amd64)
access	always "restricted"?
physicalHost	parent metal or hoster
admin	always "torproject-admin@torproject.org"
distribution	always "Debian"
l	location ("City, State, Country"), unused
ipHostNumber	IPv4 or IPv6 address, multiple values
sshRSAHostKey	SSH server public key, multiple values
rebootPolicy	how to reboot this server: manual, justdoit, rotation)

The rebootPolicy is documented in the upgrade procedures.

The purpose field is special in that it supports a crude markup language which can be used to create links in the web interface, but is also used to generate SSH known_hosts files. To quote the ud-generate source code:

In the purpose field, [[host|some other text]] (where some other text is optional) makes a hyperlink on the web [interface]. We now also add these hosts to the ssh known_hosts file. But so that we don't have to add everything we link, we can add an asterisk and say [[*... to ignore it. In order to be able to add stuff to ssh without http linking it we also support [[-hostname]] entries.

Otherwise the description and purpose fields are fairly similar and often contain the same value.

exportOptions values

• AUTHKEYS: ship the authorized_keys file for sshdist, typically on the LDAP server for ud-replicate to connect to it
• BSMTP: ship the bsmtp file
• DNS: ships DNS zone files (dns-sshfp and dns-zone)
• GITOLITE: ship the gitolite-specific SSH authorized_keys file. can also be suffixed, e.g. GITOLITE=OPTIONS where OPTIONS does magic stuff like skip some hosts (?) or change the SSH command restriction
• KEYRING: ship the sync_keyrings GnuPG keyring file (.gpg) defined in userdir-ldap.conf, generated from the admin/account-keyring.git repository (technically: the ssh://alberti.torproject.org/srv/db.torproject.org/keyrings/keyring.git repository...)
• NOMARKERS: inhibits the creation of the markers file
• NOPASSWD: if present, the passwd database has * in the password field, x otherwise. also inhibits the creation of the shadow file. also marks a host as UNTRUSTED (below)
• PRIVATE: ship the debian-private mailing list registration file
• RTC-PASSWORDS: ship the rtc-passwords file
• TOTP: ship the users.oath file
• UNTRUSTED: skip sudo passwords for this host unless explicitely set
• WEB-PASSWORDS: ship the web-passwords file

Of those parameters, only AUTHKEYS, DNS and GITOLITE are used at TPO.

Files managed by ud-generate

Path	Function	Fields used
all-accounts.json	JSON list of users	uid, uidNumber, userPassword, shadowExpire
authorized_keys	authorized_keys file for ssh_dist, if AUTHKEYS in exportOptions	ipHostNumber, sshRSAHostKey, purpose, sshdistAuthKeysHost
bsmtp	?	?
debian-private	debian-private mailing list subscription	privateSub, userPassword (skips inactive) , supplementaryGid (skips guests)
debianhosts	list of all IP addresses, unused	hostname, ipHostNumber
disabled-accounts	list of disabled accounts	uid, userPassword (includes inactive)
dns-sshfp	per-host DNS entries (e.g. debian.org), if DNS in exportOptions	see below
dns-zone	user-managed DNS entries (e.g. debian.net), if DNS in exportOptions	dnsZoneEntry
forward.alias	.forward compatibilty, unused?	uid, emailForward
group.tdb	group file template, with only the group that have access to that host	uid, gidNumber, supplementaryGid
last_update.trace	timestamp	N/A
mail-callout	?	mailCallout
mail-contentinspectionaction.cdb
mail-contentinspectionaction.db
mail-disable	disabled email messages	uid, mailDisableMessage
mail-forward.cdb	.forward "CDB" database, see cdbmake(1)	uid, emailForward
mail-forward.db	.forward Oracle Berkeley DB "DBM" database	uid, emailForward
mail-greylist	greylist the account or not	mailGreylisting
mail-rbl	?	mailRBL
mail-rhsbl	?	mailRHSBL
mail-whitelist	?	mailWhitelist
markers	xearth geolocation markers, unless NOMARKERS in extraOptions	latitude, longitude
passwd.tbd	passwd file template, if loginShell is set and user has access	uid, uidNumber, gidNumber, gecos, loginShell
rtc-passwords	secondary password for RTC calls	uid, rtcPassword, userPassword (skips inactive), supplementaryGid (skips guests)
shadow.tdb	shadow file template, same as passwd.tdb, if NOPASSWD not in extraOptions	uid, uidNumber, userPassword, shadowExpire, shadowLastChange, shadowMin, shadowMax, shadowWarning, shadowInactive
ssh-gitolite	authorized_keys file for gitolite, if GITOLITE in exportOptions	uid, sshRSAAuthKey
ssh-keys-$HOST.tar.gz	SSH user keys, as a tar archive	uid, allowed_hosts
ssh_known_host	SSH host keys	hostname, sshRSAHostKey, ipHostNumber
sudo-passwd	shadow file for sudo	uid, sudoPassword
users.oath	TOTP authentication	uid, totpSeed, userPassword (skips inactive) , supplementaryGid (skips guests)
web-passwords	secondary password database for web apps (user:pass)	uid, webPassword

File "templates" are like their regular "non-template" counterparts, except they have a prefix that corresponds to:

1. an incremental index, prefixed by zero (e.g. 01, 02, 03, ... 010...)
2. the uid field (the username), prefixed by a dot (e.g. .anarcat)
3. the uidNumber field (the UNIX UID), prefixed by an equal sign (e.g. =1092)

Those are the fields for the passwd file. The shadow file has only prefixes 1 and 2. Note that this file format is used to create the databases in /var/lib/misc/ which are fed into the NSS database with the libnss-db package. The database files get generated by makedb(1) from the templates above. It is what allows the passwd file in /etc/passwd to remain untouched while still allowing ud-ldap to manage extra users.

The authorized_keys file gets shipped if AUTHKEYS is set in extraOptions. This is typically set on the LDAP server (currently alberti), so that all servers can login to the server (as the sshdist user) and synchronise their configuration with ud-replicate. TODO: trace this to ud-replicate a little further, make a separate section?

A user gets granted access if it is part of a group that has been granted access on the host with the allowedGroups field. An additional group has access to all host, defined as allowedgroupspreload (currently adm) in /etc/userdir-ldap/userdir-ldap.conf on the LDAP server (currently alberti).

TODO: cleanup this section.

Email gateway

The email gateway runs on the LDAP server. There are four aliases, defined in /etc/aliases, which forward to the sshdist user with an extension:

change:           sshdist+changes
changes:          sshdist+changes
chpasswd:         sshdist+chpass
ping:             sshdist+ping

Then three .forward files in the ~sshdist home directory redirect this to the ud-mailgate Python program while also appending a copy of the email into /srv/db.torproject.org/mail-logs/, for example:

# cat ~sshdist/.forward+changes
"| /usr/bin/ud-mailgate change"
/srv/db.torproject.org/mail-logs/received.changes

TODO: walk through ud-mailgate.

Interactions with Puppet

The Puppet server is closely coupled with LDAP, from which it gathers information about servers.

It specifically uses those fields:

LDAP field	Puppet use
hostname	matches with the Puppet node host name, used to load records
ipHostNumber	Ferm firewall, Bind, Bacula, Jenkins, static sync access control, backends discovery
purpose	motd
physicalHost	motd: shows parent in VM, VM children in host

The ipHostnumber field is also used to lookup the host in the hoster.yaml database in order to figure out which hosting provider hosts the parent metal. This is, in turn, used in Hiera to change certain parameters, like Debian mirrors.

Note that the above fields are explicitely imported in the allnodeinfo data structure, along with sshRSAHostKey and mXRecord, but those are not used. Furthermore, the nodeinfo datastructure imports all of the host's data, so there might be other fields in use that I haven't found.

Puppet connects to the LDAP server directly over LDAPS (port 636) and therefore requires the custom LDAP host CA.

DNS zone file management

One of the configuration files ud-generate generates are, critically, the dns-sshfp and dns-zone files.

The dns-sshfp file holds the following records mapped to LDAP host fields:

DNS record	LDAP host field	Notes
SSHFP	sshRSAHostKey	extra entries possible with the sshfphostname field
A, AAAA	ipHostNumber	TTL overridable with the dnsTTL field
HINFO	architecture and machine
MX	mXRecord

The dns-zone file contains user-specific DNS entries. If a user object has a dnsZoneEntry field, that entry is written to the file directly. A TXT record with the user's email address and their PGP key fingerprint is also added for identification. That file is not in use in TPO at the moment, but is (probably?) the mechanism behind the user-editable debian.net zone.

Those files only get distributed to DNS servers (e.g. nevii and falax), which are marked with the DNS flag in the exportOptions field in LDAP.

Here is how zones are propagated from LDAP to the DNS server:

1. ud-replicate will pull the files with rsync, as explained in the previous section

2. if the dns-zone or dns-sshfp files change, ud-replicate will call /srv/dns.torproject.org/bin/update (from dns_helpers.git) as the dnsadm user, which creates the final zonefile in /srv/dns.torproject.org/var/generated/torproject.org

The bin/update script does the following:

1. pulls the auto-dns.git and domains.git git repositories

2. updates the DNSSEC keys (with bin/update-keys)

3. update the GeoIP distribution mechanism (with bin/update-geo)

4. builds the service includes from the auto-dns directory (with auto-dns/build-services), which writes the /srv/dns.torproject.org/var/services-auto/all file

5. for each domain in domains.git, calls write_zonefile (from dns_helpers.git), which in turn:

   1. increments the serial number in the .serial state file
   2. generate a zone header with the new serial number
   3. include the zone from domains.git
   4. compile it with named-compilezone(8), which is the part that expands the various $INCLUDE directives

6. then calls dns-update (from dns_helpers.git) which rewrites the named.conf snippet and reloads bind, if needed

The various $INCLUDE directives in the torproject.org zonefile are currently:

• /var/lib/misc/thishost/dns-sshfp - generated on the LDAP server by ud-generate, contains SSHFP records for each host
• /srv/dns.torproject.org/puppet-extra/include-torproject.org: generated by Puppet modules which call the dnsextras module. This is used, among other things, for TLSA records for HTTPS and SMTP services
• /srv/dns.torproject.org/var/services-auto/all: generated by the build-services script in the auto-dns.git directory
• /srv/letsencrypt.torproject.org/var/hook/snippet: generated by the bin/le-hook in the letsencrypt-domains.git repository, to authenticate against Let's Encrypt and generate TLS certificates.

Note that this procedure fails when the git server is unavailable, see issue 33766 for details.

Source file analysis

Those are the various scripts shipped by userdir-ldap. This table describes which programming language it's written in and a short description of its purpose. The ud? column documents whether the command was considered for implementation in the ud rewrite, and gives us a hint on whether it is important or not.

tool	lang	ud?	description
ud-arbimport	Python		import arbitrary entries into LDAP
ud-config	Python		prints config from userdir-ldap.conf, used by ud-replicate
ud-echelon	Python	x	"Watches for email activity from Debian Developers"
ud-fingerserv	Perl	x	finger(1) server to expose some (public) user information
ud-fingerserv2.c	C		same in C?
ud-forwardlist	Python		convert .forward files into LDAP configuration
ud-generate	Python	x	critical code path, generates all configuration files
ud-gpgimport	Python		seems unused? "Key Ring Syncronization utility"
ud-gpgsigfetch	Python		refresh signatures from a keyring? unused?
ud-groupadd	Python	x	tries to create a group, possibly broken, not implemented by ud
ud-guest-extend	Python		"Query/Extend a guest account"
ud-guest-upgrade	Python		"Upgrade a guest account"
ud-homecheck	Python		audits home directory permissions?
ud-host	Python		interactively edits host entries
ud-info	Python		same with user entries
ud-krb-reset	Perl		kerberos password reset, unused?
ud-ldapshow	Python		stats and audit on the LDAP database
ud-lock	Python	x	locks many accounts
ud-mailgate	Python	x	email operations
ud-passchk	Python		audit a password file
ud-replicate	Bash	x	rsync file distribution from LDAP host
ud-replicated	Python		rabbitmq-based trigger for ud-replicate, unused?
ud-roleadd	Python	x	like ud-groupadd, but for roles, possibly broken too
ud-sshlist	Python		like ud-forwardlist, but for ssh keys
ud-sync-accounts-to-afs	Python		sync to AFS, unused
ud-useradd	Python	x	create a user in LDAP, possibly broken?
ud-userimport	Python		imports passwd and group files
ud-xearth	Python		generates xearth DB from LDAP entries
ud-zoneupdate	Shell	x	increments serial on a zonefile and reload bind

Note how the ud-guest-upgrade command works. It generates an LDAP snippet like:

delete: allowedHost
-
delete: shadowExpire
-
replace: supplementaryGid
supplementaryGid: $GIDs
-
replace: privateSub
privateSub: $UID@debian.org

where the guest gid is replaced by the "default" defaultgroup set in the userdir-ldap.conf file.

Those are other files in the source distribution which are not directly visible to users but are used as libraries by other files.

libraries	lang	description
UDLdap.py	Python	mainly an Account representation
userdir_exceptions.py	Python	exceptions
userdir_gpg.py	Python	yet another GnuPG Python wrapper
userdir_ldap.py	Python	various functions to talk with LDAP and more

Those are the configuration files shipped with the package:

configuration files	lang	description
userdir-ldap.conf	Python	LDAP host, admin user, email, logging, keyrings, web, DNS, MX, and more
userdir_ldap.pth	???	no idea!
userdir-ldap.schema	LDAP	TPO/Debian-specific LDAP schema additions
userdir-ldap-slapd.conf.in	slapd	slapd configuration, includes LDAP access control

Issues

There is no issue tracker specifically for this project, file or search for issues in the team issue tracker, with the LDAP label.

Monitoring and testing

Logs and metrics

The LDAP directory holds a list of usernames, email addresses, real names, and possibly even physical locations. This information gets destroyed when a user is completely removed but can be kept indefinitely for locked out users.

ud-ldap keeps a full copy of all emails sent to changes@db.torproject.org, ping@torproject.org and chpass@torproject.org in /srv/db.torproject.org/mail-logs/. This includes personnally identifiable information (PII) like Received-by headers (which may include user's IP addresses), user's email addresses, SSH public keys, hashed sudo passwords, and junk mail.

TODO: expand. slapd logs? web interface?

Backups

There's no special backup procedures for the LDAP server, it is assumed that the on-disk slapd database can be backed up reliably by Bacula.

Other documentation

• userdir-ldap source code
• userdir-ldap-cgi source code
• ud - a partial ud-ldap rewrite in Django from 2013-2014, no change since 2017, the announcement for the rewrite
• userdir-ldap-pylons - a partial ud-ldap rewrite in Pylons from 2011, abandoned

Discussion

Overview

ud-ldap is decades old (the ud-generate manpage mentions 1999, but it could be older) and is hard to debug and extend. This section aims at documenting issues with the software and possible alternatives.

TODO.

Goals

Must have

Nice to have

Non-Goals

Approvals required