Skip to content
Snippets Groups Projects
Select Git revision
  • master default protected
  • tpa-rfc-80-merge-tor-and-tails-support-policies
  • puppet-merge-costs
  • rfc-77-improv
  • tails-merge-admin-mailing-lists
  • mkdocs
  • new-blog-doc
7 results
Blame Permalink
ldap.md 88.73 KiB

LDAP is a directory service we use to inventory the users, groups, passwords, (some) email forwards and machines. It distributes some configuration and password files to all machines and can reload services.

Note that this documentation needs work, particularly regarding user management procedures, see issue 40129.

Tutorial

The main LDAP documentation is on web interface. See specifically the instructions on how to:

The rest of this document is targeted at sysadmins troubleshooting LDAP issues, setting up new services, or trying to understand the setup.

Getting to know LDAP

You should have received an email like this when your LDAP account was created:

Subject: New ud-ldap account for <your name here>

That includes information about how to configure email forwarding and SSH keys. You should follow those steps to configure your SSH key to get SSH access to servers (see ssh-jump-host).

How to change my email forward?

Send an (inline!) signed OpenPGP email to changes@db.torproject.org to change your email forward. A command like this, in a UNIX shell, would do it:

echo "emailForward: user@example.com" | gpg --armor --sign

Then copy-paste that in your email client, making sure to avoid double-signing the email and sending in clear text (instead of HTML).

The email forward can also be changed in the web interface.

How-to

Set a sudo password

See the sudo password user configuration.

Know when will my change take effect?

Once a change is saved to LDAP, the actual change will take at least 5 minutes and at most 15 minutes to propagate to the relevant host. See the configuration file distribution section for more details on why it is so.

Locking an account

See the user retirement procedures.

Connecting to LDAP

LDAP is not accessible to the outside world, so you need to get behind the firewall. Most operations are done directly on the LDAP server, by logging in as a regular user on db.torproject.org (currently alberti).

Once that's resolved, you can use ldapvi(1) or ldapsearch(1) to inspect the database. User documentation on that process is in doc/accounts and https://db.torproject.org. See also the rest of this documentation.

Restoring from backups

There's no special backup procedures for the LDAP server: it's backed up like everything else in the howto/backup system.

To restore the OpenLDAP database, you need to head over the Bacula director, and enter the console:

ssh -tt bacula-director-01 bconsole

Then call the restore command and select 6: Select backup for a client before a specified time. Then pick the server (currently alberti.torproject.org) and a date. Then you need to "mark" the right files:

cd /var/lib/ldap
mark *
done

Then confirm the restore. The files will end up in /var/tmp/bacula-restores on the LDAP server.

The next step depends on whether this is a partial or total restore.

Partial restore

If you only need to access a specific field or user or part of the database, you can use slapcat to dump the database from the restored files even if the server is not running. You first need to "configure" a "fake" server in the restore directory. You will need to create two files under /var/tmp/bacula-restores:

  • /var/tmp/bacula-restores/etc/ldap/slapd.conf
  • /var/tmp/bacula-restores/etc/ldap/userdir-ldap-slapd.conf

They can be copied from /etc, with the following modifications:

diff -ru /etc/ldap/slapd.conf etc/ldap/slapd.conf
--- /etc/ldap/slapd.conf	2011-10-30 15:43:43.000000000 +0000
+++ etc/ldap/slapd.conf	2019-11-25 19:48:57.106055596 +0000
@@ -17,10 +17,10 @@
 
 # Where the pid file is put. The init.d script
 # will not stop the server if you change this.
-pidfile         /var/run/slapd/slapd.pid
+pidfile         /var/tmp/bacula-restores/var/run/slapd/slapd.pid
 
 # List of arguments that were passed to the server
-argsfile        /var/run/slapd/slapd.args
+argsfile        /var/tmp/bacula-restores/var/run/slapd/slapd.args
 
 # Read slapd.conf(5) for possible values
 loglevel        none
@@ -57,4 +57,4 @@
 #backend		<other>
 
 # userdir-ldap
-include /etc/ldap/userdir-ldap-slapd.conf
+include /var/tmp/bacula-restores/etc/ldap/userdir-ldap-slapd.conf
diff -ru /etc/ldap/userdir-ldap-slapd.conf etc/ldap/userdir-ldap-slapd.conf
--- /etc/ldap/userdir-ldap-slapd.conf	2019-11-13 20:55:58.789411014 +0000
+++ etc/ldap/userdir-ldap-slapd.conf	2019-11-25 19:49:45.154197081 +0000
@@ -5,7 +5,7 @@
 suffix          "dc=torproject,dc=org"
 
 # Where the database file are physically stored
-directory       "/var/lib/ldap"
+directory       "/var/tmp/bacula-restores/var/lib/ldap"
 
 moduleload      accesslog
 overlay accesslog
@@ -123,7 +123,7 @@
 
 
 database hdb
-directory       "/var/lib/ldap-log"
+directory       "/var/tmp/bacula-restores/var/lib/ldap-log"
 suffix cn=log
 #
 sizelimit 10000

Then slapcat is able to read those files directly:

slapcat -f /var/tmp/bacula-restores/etc/ldap/slapd.conf -F /var/tmp/bacula-restores/etc/ldap

Copy-paste the stuff you need into ldapvi.

Full rollback

Untested procedure.

If you need to roll back the entire server to this version, you first need to stop the LDAP server:

service slapd stop

Then move the files into place (in /var/lib/ldap):

mv /var/lib/ldap{,.orig}
cp -R /var/tmp/bacula-restores/var/lib/ldap /var/lib/ldap
chown -R openldap:openldap /var/lib/ldap

And start the server again:

service slapd start

Listing members of a group

To tell which users are part of a given group (LDAP or otherwise), you can use the getent(1) command. For example, to see which users are part of the tordnsel group, you would call this command:

$ getent group tordnsel
tordnsel:x:1532:arlo,arma

In the above, arlo and arma are members of the tordnsel group. The fields in the output are in the format of the group(5) file.

Note that the group membership will vary according to the machine on which the command is run, as not all users are present everywhere.

Adding/removing users in a group

Using this magical ldapvi command on the LDAP server (db.torproject.org):

ldapvi -ZZ --encoding=ASCII --ldap-conf -h db.torproject.org -D "uid=$USER,ou=users,dc=torproject,dc=org"

... you get thrown in a text editor showing you the entire dump of the LDAP database. Be careful.

To add or remove a user to/from a group, first locate that user with your editor search function (e.g. in vi, you'd type /uid=ahf to look for the ahf user). You should see a block that looks like this:

351 uid=ahf,ou=users,dc=torproject,dc=org
uid: ahf
objectClass: top
objectClass: inetOrgPerson
objectClass: debianAccount
objectClass: shadowAccount
objectClass: debianDeveloper
uidNumber: 2103
gidNumber: 2103
[...]
supplementaryGid: torproject

To add or remove a group, simply add or remove a supplementaryGid line. For example, in the above, we just added this line:

supplementaryGid: tordnsel

to add ahf to the tordnsel group.

Save the file and exit the editor. ldapvi will prompt you to confirm the changes, you can review with the v key or save with y.

Adding/removing an admin

The LDAP administrator group is a special group that is not defined through the supplementaryGid field, but by adding users into the group itself. With ldapvi (see above), you need to add a member: line, for example:

2 cn=LDAP Administrator,ou=users,dc=torproject,dc=org
objectClass: top
objectClass: groupOfNames
cn: LDAP administrator
member: uid=anarcat,ou=users,dc=torproject,dc=org

To remove the user from the admin group, remove the line.

The group grants the user access to administer LDAP directly, for example making any change through ldapvi.

Searching LDAP

This will load a text editor with a dump of all the users (useful to modify an existing user or add a new one):

ldapvi -ZZ --encoding=ASCII --ldap-conf -h db.torproject.org -D "uid=$USER,ou=users,dc=torproject,dc=org"

This dump all known hosts in LDAP:

ldapsearch -ZZ -Lx -h db.torproject.org -b "ou=hosts,dc=torproject,dc=org"

Note that this will only work on the LDAP host itself or on whitelisted hosts which are few right now. Also note that this uses an "anonymous" connection, which means that some (secret) fields might not show up. For hosts, that's fine, but if you search for users, you will need to use authentication. This, for example, will dump all users with an SSH key:

ldapsearch -ZZ -LxW -h db.torproject.org -D "uid=$USER,ou=users,dc=torproject,dc=org" -b "ou=users,dc=torproject,dc=org" '(sshRSAAuthKey=*)'

Note how we added a search filter ((sshRSAAuthKey=*)) here. We could also have parsed the output in a script or bash, but this can actually be much simpler. Also note that the previous searches dump the entire objects. Sometimes it might be useful to only list the object handles or certain fields. For example, this will list all hosts rebootPolicy attribute:

ldapsearch -h db.torproject.org -x -ZZ -b ou=hosts,dc=torproject,dc=org -LLL '(objectClass=*)' 'rebootPolicy'

This will list all servers with a manual reboot policy:

ldapsearch -h db.torproject.org -x -ZZ -b ou=hosts,dc=torproject,dc=org -LLL '(rebootPolicy=manual)' ''

Note here the empty ('') attribute list.

To list hosts that do not have a reboot policy, you need a boolean modifier:

ldapsearch -h db.torproject.org -x -ZZ -b ou=hosts,dc=torproject,dc=org -LLL '(!(rebootPolicy=manual))' ''

Such filters can be stacked to do complex searches. For example, this filter lists all active accounts:

ldapsearch -ZZ -vLxW -h db.torproject.org -D "uid=$USER,ou=users,dc=torproject,dc=org" -b "ou=users,dc=torproject,dc=org" '(&(!(|(objectclass=debianRoleAccount)(objectClass=debianGroup)(objectClass=simpleSecurityObject)(shadowExpire=1)))(objectClass=debianAccount))'

This lists users with access to Gitolite:

((allowedGroups=git-tor)|(exportOptions=GITOLITE))

... inactive users:

(&(shadowExpire=1)(objectClass=debianAccount))

Modifying the schema

If you need to add, change or remove a field in the schema of the LDAP database, it is a different, and complex operation. You will only need to do this if you launch a new service that (say) requires a new password specifically for that service.

The schema is maintained in the userdir-ldap.git repository. It is stored in the userdir-ldap.schema file. Assuming the modified object is a user, you would need to edit the file in three places:

  1. as a comment, in the beginning, to allocate a new field, for example:

    @@ -113,6 +113,7 @@
 #   .45 - rebootPolicy
 #   .46 - totpSeed
 #   .47 - sshfpHostname
+#   .48 - mailPassword
 #
 # .3 - experimental LDAP objectClasses
 #   .1 - debianDeveloper

This is purely informative, but it is important as it serves as a central allocation point for that numbering system. Also note that the entire schema lives under a branch of the Debian.org IANA OID allocation. If you reuse the OID space of Debian, it's important to submit the change to Debian sysadmins (dsa@debian.org) so they merge your change and avoid clashes.

  1. create the actual attribute, somewhere next to a similar attribute or after the previous OID, in this case we created an attributed called mailPassword right after rtcPassword, since other passwords were also grouped there:

    attributetype ( 1.3.6.1.4.1.9586.100.4.2.48
       NAME 'mailPassword'
       DESC 'mail password for SMTP'
       EQUALITY octetStringMatch
       SYNTAX 1.3.6.1.4.1.1466.115.121.1.40 )

  2. finally, the new attribute needs to be added to the objectclass. in our example, the field was added alongside the other password fields in the debianAccount objectclass, which looked like this after the change:

    objectclass ( 1.3.6.1.4.1.9586.100.4.1.1
	NAME 'debianAccount'
	DESC 'Abstraction of an account with POSIX attributes and UTF8 support'
	SUP top AUXILIARY
	MUST ( cn $ uid $ uidNumber $ gidNumber )
	MAY ( userPassword $ loginShell $ gecos $ homeDirectory $ description $ mailDisableMessage $ sudoPassword $ webPassword $ rtcPassword $ mailPassword $ totpSeed ) )

Once that schema file is propagated to the LDAP server, this should automatically be loaded by slapd when it is restarted (see below). But the ACL for that field should also be modified. In our case, we had to add the mailPassword field to two ACLs:

--- a/userdir-ldap-slapd.conf.in
+++ b/userdir-ldap-slapd.conf.in
@@ -54,7 +54,7 @@ access to attrs=privateSub
        by * break
 
 # allow users write access to an explicit subset of their fields
-access to attrs=c,l,loginShell,ircNick,labeledURI,icqUIN,jabberJID,onVacation,birthDate,mailDisableMessage,gender,emailforward,mailCallout,mailGreylisting,mailRBL,mailRHSBL,mailWhitelist,mailContentInspectionAction,mailDefaultOptions,facsimileTelephoneNumber,telephoneNumber,postalAddress,postalCode,loginShell,onVacation,latitude,longitude,VoIP,userPassword,sudoPassword,webPassword,rtcPassword,bATVToken
+access to attrs=c,l,loginShell,ircNick,labeledURI,icqUIN,jabberJID,onVacation,birthDate,mailDisableMessage,gender,emailforward,mailCallout,mailGreylisting,mailRBL,mailRHSBL,mailWhitelist,mailContentInspectionAction,mailDefaultOptions,facsimileTelephoneNumber,telephoneNumber,postalAddress,postalCode,loginShell,onVacation,latitude,longitude,VoIP,userPassword,sudoPassword,webPassword,rtcPassword,mailPassword,bATVToken
        by self write
        by * break
 
@@ -64,7 +64,7 @@ access to attrs=c,l,loginShell,ircNick,labeledURI,icqUIN,jabberJID,onVacation,bi
 ##
 
 # allow authn/z by anyone
-access to attrs=userPassword,sudoPassword,webPassword,rtcPassword,bATVToken
+access to attrs=userPassword,sudoPassword,webPassword,rtcPassword,mailPassword,bATVToken
        by * compare
 
 # readable only by self

If those are the only required changes, it is acceptable to directly make those changes directly on the LDAP server, as long as the exact same changes are performed in the git repository.

It is preferable, however, to build and upload userdir-ldap as a Debian package instead.

Deploying new userdir-ldap releases

Our userdir-ldap codebase is deployed through Debian packages built by hand on TPA's members computers, from our userdir-ldap repository. Typically, when we make changes to that repository, we should make sure we send the patches upstream, to the DSA userdir-ldap repository. The right way to do that is to send the patch by email, to mailto:dsa@debian.org, since they do not have merge requests enabled on that repository.

If you are lucky, we will have the latest version of the upstream code and your patch will apply cleanly upstream. If unlucky, you'll actually need to merge with upstream first. This process is generally done through those steps:

  1. git merge the upstream changes, and resolve the conflicts
  2. update the changelog (make sure you have the upstream version with ~tpo1 as a suffix so that upgrades work when if we ever catch up with upstream)
  3. build the Debian package: git buildpackage
  4. deploy the Debian package

Note that unless the change is trivial, the Debian package should be deployed very carefully. Because userdir-ldap is such a critical piece of infrastructure, it can easily break stuff like PAM and logins, so it is important to deploy it one machine at a time, and run ud-replicate on the deployed machine (and ud-generate if the machine is the LDAP server).

So "deploy the Debian package" should actually be done by copying, by hand, the package to specific servers over SSH, and only after testing there, uploading it to the Debian archive.

Note that it's probably a good idea to update the userdir-ldap-cgi repository alongside userdir-ldap. The above process should similarly apply.

Pager playbook

An LDAP server failure can trigger lots of emails as ud-ldap fails to synchronize things. But the infrastructure should survive the downtime, because users and passwords are copied over to all hosts. In other words, authentication doesn't rely on the LDAP server being up.

In general, OpenLDAP is very stable and doesn't generally crash, so we haven't had many emergencies scenarios with it yet. If anything happens, make sure the slapd service is running.

The ud-ldap software, on the other hand, is a little more complicated and can be hard to diagnose. It has a large number of moving parts (Python, Perl, Bash, Shell scripts) and talks over a large number of protocols (email, DNS, HTTPS, SSH, finger). The failure modes documented here are far from exhaustive and you should expect exotic failures and error messages.

LDAP server failure

That said, if the LDAP server goes down, password changes will not work, and the server inventory (at https://db.torproject.org/) will be gone. A mitigation is to use Puppet manifests and/or PuppetDB to get a host list and server inventory, see the Puppet documentation for details.

Git server failure

The LDAP server will fail to regenerate (and therefore update) zone files and zone records if the Git server is unavailable. This is described in issue 33766. The fix is to recover the git server. A workaround is to run this command on the primary DNS server (currently nevii):

sudo -u dnsadm /srv/dns.torproject.org/bin/update --force

Deadlocks in ud-replicate

The ud-replicate process keeps a "reader" lock on the LDAP server. If for some reason the network transport fails, that lock might be held on forever. This happened in the past on hosts with flaky network or ipsec problems that null-routed packets between ipsec nodes.

There is a Nagios check that will detect stale synchronisations. Example:

Subject: ** PROBLEM Service Alert: palmeri/setup - ud-ldap freshness is WARNING **

Note that this can generate a lot of warnings because one per server will be sent!

The fix is to find the offending locked process and kill it. In desperation:

pkill -u sshdist rsync

... but really, you should carefully review the rsync processes before killing them all like that. And obviously, fixing the underlying network issue would be important to avoid such problems in the future.

Also note that the lock file is in /var/cache/userdir-ldap/hosts/ud-generate.lock, and ud-generate tries to get a write lock on the file. This implies that a deadlock will also affect file generation and keep ud-generate from generating fresh config files.

Finally, ud-replicate also holds a lock on /var/lib/misc on the client side, but that rarely causes problems.

Troubleshooting changes@ failures

A common user question is that they are unable to change their SSH key. This can happen if their email client somehow has trouble sending a PGP signature correctly. Most often than not, this is because their email client does a line wrap or somehow corrupts the OpenPGP signature in the email.

A good place to start looking for such problems is the log files on the LDAP server (currently alberti). For example, this has a trace of all the emails received by the changes@ alias:

/srv/db.torproject.org/mail-logs/received.changes

A common problem is people using --clearsign instead of --sign when sending an SSH key. When that happens, many email clients (including Gmail) will word-wrap the SSH key after the comment, breaking the signature. For example, this might happen:

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKxqYYEeus8dRXBHhLsp0SjH7ut2X8UM9hdXN=
wJIl89otcJ5qKoXj90K9hq8eBjG2KuAZtp0taGQHqzBOFK+sFm9/gIqvzzQ07Pn0xtkmg10Hunq=
vPKMj4gDFLIqTF0WSPA2E6L/TWaeVJ+IiGuE49j+0Ohd7UFDEquM1H/zno22vIEm/dxWLPWD9gG=
MmwBghvfK/dRyzSEDGlAVeWLzoIvVOG12/ANgic3TlftbhiLKTs52hy8Qhq/aQBqd0McaE4JGxe=
9k71OCg+0WHVS4q7HVdTUqT3VFFfz0kjDzYTYQQcHMqPHvYzZghxMVCmteNdJNwJmGSNPVaUeJG=
MumJ9
anarcat@curie

-----BEGIN PGP SIGNATURE-----
[...]
-----END PGP SIGNATURE-----

Using --sign --armor will work around this problem, as the original message will all be ASCII-armored.

Dependency loop on new installs

Installing a new server requires granting the new server access various machines, including puppet and the LDAP server itself. This is granted ... by Puppet through LDAP!

So a server cannot register itself on the LDAP server and needs an operator to first create a host snippet on the LDAP server, and then run Puppet on the Puppet server. This is documented in the installation notes.

Disaster recovery

The LDAP server is mostly built by hand and should therefore be restored from backups in case of a catastrophic failure. Care should be taken to keep the SSH keys of the server intact.

The IP address (and name?) of the LDAP server should not be hard-coded anywhere. When the server was last renumbered (issue 33908), the only changes necessary were on the server itself, in /etc. So in theory, a fresh new server could be deployed (from backups) in a new location (and new address) without having to do much.

Reference

Installation

All ud-ldap components are deployed through Debian packages, compiled from the git repositories. It is assumed that some manual configuration was performed on the main LDAP server to get it bootstrapped, but that procedure was lost in the mists of time.

Only backups keep us from total catastrophe in case of lost. Therefore, this system probably cannot be reinstalled from scratch.

SLA

The LDAP server is designed to be fault-tolerant in the sense that it's database is copied over other hosts. It should otherwise be highly available as it's a key component in managing users authentication and authorization, and machines.

Design

The LDAP setup at Tor is based on the one from Debian.org. It has a long, old and complex history, lost in the mists of time.

Configuration and database files like SSH keys, OpenPGP keyrings, password, group databases, or email forward files are synchronised to various hosts from the LDAP database. Most operations can be performed on the db.torproject.org site or by email.

Architecture overview

This is all implemented by a tool called ud-ldap, inherited from the Debian project. The project is made of a collection of bash, Python and Perl scripts which take care of synchronizing various configuration files to hosts based on the LDAP configuration. Most of this section aims at documenting how this program works.