Unify Metrics' products operational configuration

The method/process of deployment configuration of Metrics' products should be unified. Afterwards, we look for the 'tool' or lib helping to achieve a simplification.

---

The thought that triggered this ticket: Looks like metrics-bot recently switched to using Apache Commons Configuration as configuration library (#23933 (closed)). We should consider doing the same for other metrics parts, maybe after waiting a bit and hearing how that works out.

added component::metrics metrics-team-roadmap-2020 owner::metrics-team points::2 priority::medium severity::normal status::assigned type::enhancement labels

Maybe rather summarize this ticket as: "Unify Metrics' products operational configuration"

The method/process of deployment configuration of Metrics' products should be unified. Afterwards, we look for the 'tool' or lib helping to achieve a simplification.

---

Some thoughts:

It seems that from all options (hard-coding, command-line parameters, and config files) the config file solution is best. Also, because a configuration file can be checked into a local git on the deployment machine, i.e., less guessing what parameters are needed/ were used.

A configuration file should follow a format/standard. Java properties should be first choice because they are easy to understand and Java provides the appropriate functionality for accessing them. Second choice to make is between simple key-value properties and XML properties. IMO, the simple format ought to be sufficient and is better for 'human consumption'.

CollecTor features the most complex configuration settings of all Metrics' products and is still based on simple key-value Java properties. All other products (cf. list below) have less configuration demands that could all be met by Java properties.

Tools like commons-configuration provide 99% functionality that is beyond the needs of Metrics' products. Using the functionality from there does not really simplify the code compared to using jdk-provided methods.

Suggestion:

1. Use key value properties files for configuration.
2. Use Java provided functionality for accessing the properties.
3. If there is the need for more sophisticated configuration methods throughout several products, look to provide them in a standardized way (i.e., in a metrics-lib's util package).

---

Overview of the current state as reference:

• Onionoo: command line settings and hard coded settings
• CollecTor: configuration file and Metrics' code for configuration checks and convenience methods incl. provision of a basic config file. Using Java properties format.
• ExoneraTor: configuration file read as a file (back end) and system properties (web part).
• Web: mixture off all.
• Bot: Apache Commons Configuration as configuration library (#23933 (closed)).

Agreed, this is what I meant. No need to add another dependency if we can solve the problem with the JDK alone.

irl, what was the reason that you switched to Apache Commons Configuration? What parts do you use that are harder or impossible to do with Java properties?

Changing the ticket's summary and description a little to reflect the actual topic.

Trac:
Summary: Add Apache Commons Configuration as configuration library to Unify Metrics' products operational configuration
Description: Looks like metrics-bot recently switched to using Apache Commons Configuration as configuration library (#23933 (closed)). We should consider doing the same for other metrics parts, maybe after waiting a bit and hearing how that works out.

to

The method/process of deployment configuration of Metrics' products should be unified. Afterwards, we look for the 'tool' or lib helping to achieve a simplification.

---

The thought that triggered this ticket: Looks like metrics-bot recently switched to using Apache Commons Configuration as configuration library (#23933 (closed)). We should consider doing the same for other metrics parts, maybe after waiting a bit and hearing how that works out.

Trac:
Owner: metrics-team to iwakeh
Status: new to accepted

Java properties didn't really provide the hierarchical structure to the configuration that is needed for the bot. There are types of accounts, and then there can be 0 or more of each type, and then those have properties within them.

An example XML file is now included in the documentation:

https://people.torproject.org/~irl/metrics-bot/org/torproject/metrics/bot/Configuration.html

If all you need are strings and lists then properties is fine, but once you get into lists of lists it gets a lot more difficult and you have to start working out your own syntax for separating items, etc.

Commons Configuration is in Debian as libcommons-configuration-java (1.10 in both testing and stable).

metrics-team was not added to cc

Trac:
Cc: N/A to metrics-team

Sounds like a good topic for the next team meeting.

Most Metrics' products, like Onionoo, Exonerator, and Web don't even have the single file as configuration option (cf. comment:1 overview).

It seems we all agree on the goal to have one file that configures each product's operational options.

Both Bot and CollecTor are at that stage. Onionoo, Exonerator, and Web need to be 'pushed' to that level of configuration, which will be more work in making the various settings accessible in one configuration file. AfaIk, simple Java properties suffice, which could come in either properties layout or simple XML; and could be either implemented in plain Java or using commons configuration.

The focus is the operational advantage, which will consist in having the 'single point of configuration' and we should just make sure to choose the same kind of implementation approach for Onionoo, Exonerator, and Web; Bot and CollecTor work as they are. Implementing the configuration choice for these two is a secondary (more long-term) issue, because both work fine as they are.

Another question is what all should be configurable? I.e., this ticket should be also used for checking what values, e.g., working directories, base URLs etc., ought to be configurable and are maybe are not yet.

Trac:
Cc: metrics-team to N/A
Status: accepted to assigned
Owner: iwakeh to metrics-team

Trac:
Actualpoints: N/A to 6
Keywords: N/A deleted, metrics-team-roadmap-2020April added
Points: N/A to 8

Trac:
Points: 8 to 2
Actualpoints: 6 to N/A

Trac:
metrics-configuration-overview-2020-04-12.pdf

I made some progress here by creating an overview of configuration options in our various code bases. This isn't really ready for review yet, but I'm planning to do something else next week, so I'm putting it here for picking it up later. It's the attached PDF.

I also made a couple comments while putting together this overview:

• It would be nice to have a single naming scheme for options. Right now we have ALLCAPITAL, CamelCase, dotted.notation, and --two-dashes.
• It would be nice to have a single way of specifying options, rather than having CollecTor's collector.properties file, ExoneraTor‘s config file, Onionoo‘s command line arguments, and some -Djava.properties.
• The metrics-web updater is lacking options to disable or select single modules. Right now this requires a temporary code change.
• The metrics-web updater is also lacking an option to disable just the module that fetches descriptors from CollecTor. Right now we're removing a line in the source code when running integration tests.
• The Onionoo updater is lacking an option to disable rDNS lookups, which is also something where we're removing some code when running integration tests.

We need to review all this tickets for metrics roadmap.

Trac:
Keywords: metrics-team-roadmap-2020April deleted, metrics-team-roadmap-2020 added

changed time estimate to 16h

mentioned in issue #24174 (moved)

mentioned in issue #24328 (moved)

mentioned in issue #25700 (moved)

mentioned in issue #25799 (moved)