document Puppet/LDAP integration

howto/puppet.md

@@ -708,6 +708,59 @@ neded more functions (like `map` and `filter`) to get what I wanted
 (see [this gist](https://gist.github.com/bastelfreak/b9620fa1892ebcc659c442b115db34f9)). I gave up at that point: the `puppetdbquery`
 abstraction is much cleaner and usable.
 
+If you are merely looking for a hostname, however, PQL might be a
+little more manageable. For example, this is how the
+`roles::onionoo_frontend` class finds its backends to setup the
+[IPsec](ipsec) network:
+
+    $query = 'nodes[certname] { resources { type = "Class" and title = "Roles::Onionoo_backend" } }'
+    $peer_names = sort(puppetdb_query($query).map |$value| { $value["certname"] })
+    $peer_names.each |$peer_name| {
+      $network_tag = [$::fqdn, $peer_name].sort().join('::')
+      ipsec::network { "ipsec::${network_tag}":
+        peer_networks => $base::public_addresses
+      }
+    }
+
+### LDAP lookups
+
+Our Puppet server is hooked up to the LDAP server and has information
+about the hosts defined there. Information about the node running the
+manifest is available in the global `$nodeinfo` variable, but there is
+also a `$allnodeinfo` parameter with information about every host
+known in LDAP.
+
+A simple example of how to use the `$nodeinfo` variable is how the
+`base::public_address` and `base::public_address6` parameters -- which
+represent the IPv4 and IPv6 public address of a node -- are
+initialized in the `base` class:
+
+    class base(
+      Stdlib::IP::Address $public_address            = filter_ipv4(getfromhash($nodeinfo, 'ldap', 'ipHostNumber'))[0],
+      Optional[Stdlib::IP::Address] $public_address6 = filter_ipv6(getfromhash($nodeinfo, 'ldap', 'ipHostNumber'))[0],
+    ) {
+      $public_addresses = [ $public_address, $public_address6 ].filter |$addr| { $addr != undef }
+    }
+
+This loads the `ipHostNumber` field from the `$nodeinfo` variable, and
+uses the `filter_ipv4` or `filter_ipv6` functions to extract the IPv4
+or IPv6 addresses respectively.
+
+A good example of the `$allnodeinfo` parameter is how the
+`roles::onionoo_frontend` class finds the IP addresses of its
+backend. After having loaded the host list from PuppetDB, it then uses
+the parameter to extract the IP address:
+
+    $backends = $peer_names.map |$name| {
+        [
+          $name,
+          $allnodeinfo[$name]['ipHostNumber'].filter |$a| { $a =~ Stdlib::IP::Address::V4 }[0]
+        ] }.convert_to(Hash)
+
+Such a lookup is considered more secure than going through PuppetDB as
+LDAP is a trusted data source. It is also our source of truth for this
+data, at the time of writing.
+
 ### Hiera lookups
 
 For more security-sensitive data, we should use a trusted data source
@@ -733,10 +786,6 @@ could be overriden per host. Similarly, the IP address information for
 each host could be stored in Hiera instead of LDAP. But in practice,
 we do not currently do this and the per-host information is limited.
 
-### LDAP lookups
-
-TODO.
-
 ## Revoking and generating a new certificate for a host
 
 Revocation procedures problems were discussed in [33587][] and [33446][].
@@ -842,13 +891,22 @@ backing the PuppetDB server as well. It's *possible* this step *could*
 be skipped in an emergency, because most of the information in
 PuppetDB is a cache of exported resources, reports and facts. But it
 could also break hosts and make converging the infrastructure
-impossible, as there might be dependency loops in exported resources
-(for example, the Puppet server needs access to the LDAP server, and
-that is configured in Puppet). 
+impossible, as there might be dependency loops in exported resources.
+
+In particular, the Puppet server needs access to the LDAP server, and
+that is configured in Puppet. So if the Puppet server needs to be
+rebuilt from scratch, it will need to be manually allowed access to
+the LDAP server to compile its manifest.
 
 So it is strongly encouraged to restore the PuppetDB server database
 as well in case of disaster.
 
+This also applies in case of an IP address change of the Puppet
+server, in which case access to the LDAP server needs to be manually
+granted before the configuration can run and converge. This is a known
+bootstrapping issue with the Puppet server and is further discussed in
+the [design section](#LDAP-integration).
+
 # Reference
 
 This documents generally how things are setup.
@@ -1072,10 +1130,51 @@ Puppet itself, currently as part of the `torproject_org` module.
 
 ### LDAP integration
 
-TODO: document how Puppet talks with LDAP (and vice-versa?). Note that
-this is from a design perspective (ie. firewalls, access controls,
-passwords, etc), not from a "user" perspective (ie. how to actually do
-it in the Puppet code).
+The Puppet is configured to talk with Puppet through a few custom
+functions defined in
+`modules/puppetmaster/lib/puppet/parser/functions`. The main plumbing
+function is called `ldapinfo()` and connects to the LDAP server
+through `db.torproject.org` over TLS on port 636. It takes a hostname
+as an argument and will load all hosts matching that pattern under the
+`ou=hosts,dc=torproject,dc=org` subtree. If the specified hostname is
+the `*` wildcard, the result will be a hash of `host => hash` entries,
+otherwise only the `hash` describing the provided host will be
+returned.
+
+The `nodeinfo()` function uses that function to populate the global
+`$nodeinfo` hash available globally, or, more specifically, the
+`$nodeinfo['ldap']` component. It also loads the `$nodeinfo['hoster']`
+value from the `whohosts()` function. That function, in turn, tries to
+match the IP address of the host against the "hosters" defined in the
+`hoster.yaml` file.
+
+The `allnodeinfo()` function does a similar task as `nodeinfo()`,
+except that it loads *all* nodes from LDAP, into a single hash. It
+does *not* include the "hoster" and is therefore equivalent to calling
+`nodeinfo()` on each host and extracting only the `ldap` member hash
+(although it is not implemented that way).
+
+Puppet does not require any special credentials to access the LDAP
+server. It accesses the LDAP database anonymously, although there is a
+firewall rule (defined in Puppet) that grants it access to the LDAP
+server. 
+
+There is a bootstrapping problem there: if one would be to rebuild the
+Puppet server, it would actually fail to compile its catalog because
+it would not be able to connect to the LDAP server to fetch its
+catalog, unless the LDAP server has been manually configured to let
+the Puppet server through.
+
+NOTE: much (if not all?) of this is being moved into Hiera, in
+particular the YAML files. See [issue 30020](https://trac.torproject.org/projects/tor/ticket/30020) for details. Moving
+the host information into Hiera would resolve the bootstrapping
+issues, but would require, in turn some more work to resolve questions
+like how users get granted access to individual hosts, which is
+currently managed by `ud-ldap`. We cannot, therefore, simply move host
+information from LDAP into Hiera without creating a duplicate source
+of truth without rebuilding or tweaking the user distribution
+system. See also the [LDAP design document](ldap#Design) for more information
+about how LDAP works.
 
 ### External data sources