Proposal: Centralized Probe Log Collection
Background
Currently, there are two vantage points that run automated reachability probetest. However, these probe results are not automatically collected and available for analysis. This made it more difficult to monitor the result of these probes and get alert for censorship events.
Proposal
Use a centralized log collector to receive important data collected from different locations. The communication will be conducted in standardized gRPC requests(or plain HTTP if preferred).
The log collector client will read the tar file generated by probetest, and extract the important data from it to create a summary. This summary will be then be uploaded to the server for further processing.
The log collector server will collect probe results summaries submitted by the clients, and ingest this result into different forms, like an archival log, Prometheus(and possibly alerts).
Tasks
- Design the summary format (Related: censorship-analysis#40022)
- Write client side summary extractor
- Write server side summary receiver
- Write client side summary submission utility
- Write server side summary archival log ingester
- Write server side summary to Prometheus ingester
Design considerations
- By submitting all probe results to a single server, probe result data will be able to survive any outage in vantage points. This will also make it possible to easily add any new vantage points without the need for setting modifications on the data collection server. This also makes it possible to reduce the resource usage on vantage point device by reducing the need for running software like Prometheus on the vantage point.
- By using gRPC as a transport method, the boilerplate code for processing RPC related content is minimalized.
- By supporting more than one ingesting routine, additional modes of operation can be added without interference with existing codes. (It will use something similar to the event interface designed for snowflake)
- The summary format is defined as protocol buffer structure and represented as protojson. This allows it to have a strictly defined schema.