The Tor Project issueshttps://gitlab.torproject.org/groups/tpo/-/issues2021-08-23T13:18:03Zhttps://gitlab.torproject.org/tpo/network-health/metrics/collector/-/issues/40002CollecTor should archive sanitised bridgestrap results2021-08-23T13:18:03ZirlCollecTor should archive sanitised bridgestrap resultsTo monitor the health of bridges in the network over time, and also to support providing better information to bridge operators via Onionoo/Relay Search, CollecTor should archive sanitised bridgestrap results.
c.f. https://gitlab.torpro...To monitor the health of bridges in the network over time, and also to support providing better information to bridge operators via Onionoo/Relay Search, CollecTor should archive sanitised bridgestrap results.
c.f. https://gitlab.torproject.org/tpo/metrics/relay-search/-/issues/40003irlirl2021-07-31https://gitlab.torproject.org/tpo/onion-services/onion-support/-/issues/109Write a script to gather EOTK stats from logs hosted on AWS S3 buckets2022-06-03T12:44:53ZSilvio RhattoWrite a script to gather EOTK stats from logs hosted on AWS S3 bucketsWrite a script to gather EOTK stats. Related to [this Bypass Censorship Dashboard issue](https://gitlab.com/guardianproject/bypass-censorship/analytics-dashboard/-/issues/1).Write a script to gather EOTK stats. Related to [this Bypass Censorship Dashboard issue](https://gitlab.com/guardianproject/bypass-censorship/analytics-dashboard/-/issues/1).Sponsor 123: Tor Secure Access Package for USAGM [First Phase]Silvio RhattoSilvio Rhatto2022-06-08https://gitlab.torproject.org/tpo/tpa/team/-/issues/40814OOM issue on meronense after upgrade2024-02-02T03:23:35ZHiroOOM issue on meronense after upgradeNoticed metrics.tpo is not getting all its updates since postgresql has been upgraded to v13.
I have started the script manually: https://gitlab.torproject.org/tpo/network-health/metrics/metrics-bin/-/blob/main/website/run-web.sh
And f...Noticed metrics.tpo is not getting all its updates since postgresql has been upgraded to v13.
I have started the script manually: https://gitlab.torproject.org/tpo/network-health/metrics/metrics-bin/-/blob/main/website/run-web.sh
And found out the job was being killed:
```
[308908.109696] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/user.slice/user-0.slice/session-4020.scope,task=java,pid=375579,uid=1512
[308908.109723] Out of memory: Killed process 375579 (java) total-vm:14411748kB, anon-rss:7917568kB, file-rss:0kB, shmem-rss:32kB, UID:1512 pgtables:23120kB oom_score_adj:0
```
cc: @gkanarcatanarcat2022-07-27https://gitlab.torproject.org/tpo/onion-services/onion-support/-/issues/99General policy on how the Onion Support team should respond to Onionprobe alerts2022-09-27T09:49:18ZSilvio RhattoGeneral policy on how the Onion Support team should respond to Onionprobe alertsWrite a small quick policy in what to do when an incident happen and are detected/notified by Onionprobe, including:
* Inform the interested parties (like Sponsors and/or users) of the issue, if needed.
* Check agreement when/where admi...Write a small quick policy in what to do when an incident happen and are detected/notified by Onionprobe, including:
* Inform the interested parties (like Sponsors and/or users) of the issue, if needed.
* Check agreement when/where admins allow to be notified.
* Then check if admins online (or on shift) can work on it (depends on agreed channels and current time):
* Ping on IRC.
* Ping on email.
* Ping on Signal.
* Ping on X.
This policy should be available at the Onion Support wiki.Sponsor 123: Tor Secure Access Package for USAGM [First Phase]Silvio RhattoSilvio Rhatto2022-08-31https://gitlab.torproject.org/tpo/onion-services/onion-launchpad/-/issues/39Create a privacy-preserving landing page analytics roadmap for S1232022-11-23T15:08:02ZrayaCreate a privacy-preserving landing page analytics roadmap for S123Following on from the discussion which started in the July 2022 narrative report [ticket](https://gitlab.torproject.org/tpo/onion-services/onion-support/-/issues/130#note_2823857), we need to discuss next steps for implementing a method ...Following on from the discussion which started in the July 2022 narrative report [ticket](https://gitlab.torproject.org/tpo/onion-services/onion-support/-/issues/130#note_2823857), we need to discuss next steps for implementing a method to collect the number of visitors per S123 landing page.
Copy-pasting the 4 steps suggested by @rhatto to plug Clean Insights (which is based on Matomo and built by The Guardian Project):
> 1. Frontend modifications into the existing landing page code to plug [Clean Insights JS SDK](https://gitlab.com/cleaninsights/clean-insights-js-sdk).
> 2. Consider that this will need some [consent UX](https://okthanks.com/blog/2021/5/14/clean-consent-ux).
> 3. Setting up [Clean Insights Infrastructure](https://gitlab.com/cleaninsights/clean-insights-infrastructure) in a new virtual machine (and maintaining that machine).
> 4. Making sure that landing page deployments from the same service always use the same `siteId` (so stats will be gathered no matter how many mirrors exists and where they're hosted).
I made the issue confidential but I don't believe it needs to be!
cc: @rhattoSponsor 123: Tor Secure Access Package for USAGM [First Phase]Silvio RhattoSilvio Rhatto2022-11-30https://gitlab.torproject.org/tpo/onion-services/onion-support/-/issues/162Get EOTK stats for Sponsor 123 during November 20222022-12-08T19:28:07ZSilvio RhattoGet EOTK stats for Sponsor 123 during November 2022* [x] Get statistics for the November 2022 Narrative Report as [per contract](https://nc.torproject.net/apps/onlyoffice/242116?filePath=%2FSponsors%2FS123%20-%20USAGM%2FSubmitted%20Documents%2F2021-09-17%20Tor%20Secure%20Access%20-%20OTF...* [x] Get statistics for the November 2022 Narrative Report as [per contract](https://nc.torproject.net/apps/onlyoffice/242116?filePath=%2FSponsors%2FS123%20-%20USAGM%2FSubmitted%20Documents%2F2021-09-17%20Tor%20Secure%20Access%20-%20OTF%20TaS%20Narrative-1-20-2022_203_PM.docx):
* [x] Uptime of USAGM .onion addresses.
* [x] Number of visitors to USAGM .onion addresses (page hits, if `HiddenServiceExportCircuitID` data is not available).
* [x] Make sure that Onionprobe data is filtered out.
* [x] [Progress Tracker](https://nc.torproject.net/apps/onlyoffice/296583?filePath=%2FOnion%20Services%2FOnion%20Support%2FS123%20Progress%20Tracker.ods) (or a standalone sheet): statistics subsheet built from [eotk-log-parser](https://gitlab.torproject.org/tpo/onion-services/eotk-log-parser).
* [x] Create the ticket for the next stats gathering.
/cc @rayaSponsor 123: Tor Secure Access Package for USAGM [First Phase]Silvio RhattoSilvio Rhatto2022-12-01https://gitlab.torproject.org/tpo/onion-services/onion-launchpad/-/issues/66Frontend support for Matomo analytics2023-05-16T14:34:32ZSilvio RhattoFrontend support for Matomo analyticsOnion Launchpad implementation of Matomo analytics according to [this proposal][].
## Requirements
1. [x] The feature MUST be *disabled* by default, and enabled only if some
environment variables are set (like the analytics endp...Onion Launchpad implementation of Matomo analytics according to [this proposal][].
## Requirements
1. [x] The feature MUST be *disabled* by default, and enabled only if some
environment variables are set (like the analytics endpoint and a site ID/key).
2. [x] There MUST be documentation stating that this feature, even with a better
configuration in terms of privacy, could still be a point of collecting
access data without passing to the Tor network for better anonymization. And
also would rely on additional JavaScript code embedded in the landing page.
3. [x] Services operators MUST be recommended to host the backend only behind an
HTTPS proxy without IP logging (and without passing the source IP to the
backend, so if there's any backend vulnerability it won't be possible to
attackers to discover user's IP addresses). ___Or even better: leave the
backend behind an Onion Service___.
4. [x] There MUST be a [consent UX][] informing users what and how it's collected,
and asking for authorization. No cookies should reside in the client machine.
## Implementation details
* [x] Plug the [Clean Insights JS SDK][] or the [Matomo JS SDK][] into [Onion Launchpad][].
* [x] Enable the metrics collection only if explicitly set by an environment variable during build time.
* [x] Implement a [consent UX][].
* [x] Implement the page hits collection.
* [x] Document the [analytics collection threat model][] (subsection "Landing page metrics" of this link/comment).
[analytics collection threat model]: https://gitlab.torproject.org/tpo/onion-services/onion-launchpad/-/issues/39#note_2854486
## Scope
This ticket covers:
* Basic functionality without censorship protection for the metrics system.
* Description: in this phase, the whole implementation is completed.
This ticket does not cover:
* The backend development.
* Implementing censorship protection for the metrics system.
* Content and styling for the consent UX (handled in a [distinct ticket][]).
[this proposal]: https://gitlab.torproject.org/tpo/onion-services/onion-launchpad/-/issues/39#note_2854486
[Onion Launchpad]: https://gitlab.torproject.org/tpo/onion-services/onion-launchpad
[distinct ticket]: tpo/onion-services/onion-launchpad#67
[Clean Insights JS SDK]: https://gitlab.com/cleaninsights/clean-insights-js-sdk
[Matomo JS SDK]: https://developer.matomo.org/guides/tracking-javascript-guide
[consent UX]: https://okthanks.com/blog/2021/5/14/clean-consent-uxSponsor 123: Tor Secure Access Package for USAGM [First Phase]Silvio RhattoSilvio Rhatto2023-01-20https://gitlab.torproject.org/tpo/onion-services/onionprobe/-/issues/49Tests and metrics for X.509 certificates2023-06-14T20:48:53ZSilvio RhattoTests and metrics for X.509 certificatesOnionprobe should check and have metrics for X.509 certificates:
* [x] Expiry date.
* [x] Fingerprint.
* [x] Issuer.
It also needs:
* [x] CommonName and SubjectAltName matching.
* [x] A Prometheus alerts for certificates about to expi...Onionprobe should check and have metrics for X.509 certificates:
* [x] Expiry date.
* [x] Fingerprint.
* [x] Issuer.
It also needs:
* [x] CommonName and SubjectAltName matching.
* [x] A Prometheus alerts for certificates about to expire.
That enables monitoring of certificate health and even alerting when certificates are about to expire.Silvio RhattoSilvio Rhatto2023-03-31https://gitlab.torproject.org/tpo/onion-services/onionprobe/-/issues/74Make it easier to setup Tor's MetricsPort2023-03-30T18:27:51ZSilvio RhattoMake it easier to setup Tor's MetricsPortRight now there are a number of files to be edited in order to have Tor metrics into Prometheus.
Make it easier to setup `MetricsPort` at the standalone monitoring node.Right now there are a number of files to be edited in order to have Tor metrics into Prometheus.
Make it easier to setup `MetricsPort` at the standalone monitoring node.Sponsor 123: Tor Secure Access Package for USAGM [First Phase]Silvio RhattoSilvio Rhatto2023-04-03https://gitlab.torproject.org/tpo/onion-services/onionprobe/-/issues/71Tor metrics Prometheus exporter2023-03-30T17:04:06ZSilvio RhattoTor metrics Prometheus exporterSetup a Tor metrics Prometheus exporter, so Onionprobe could have two distinct
Prometheus exporters:
1. The default, existing one with Onionprobe metrics.
2. Another with the Tor process metrics.
Tasks:
* [x] Add an Option to setup a ...Setup a Tor metrics Prometheus exporter, so Onionprobe could have two distinct
Prometheus exporters:
1. The default, existing one with Onionprobe metrics.
2. Another with the Tor process metrics.
Tasks:
* [x] Add an Option to setup a `MetricsPort` and `MetricsPortPolicy`
on the spawned Tor process, but disabled by default.
* [x] Include it also on the Prometheus collection and make it
available as a Grafana dashboard, but disabled by default.
* [x] Add a warning/document somewhere that this setting should be used with care.
* [x] Document how to enable the feature in the standalone monitoring node,
including again the warnings about this setting.
Documentation: https://gitlab.torproject.org/tpo/core/tor/-/issues/40762Sponsor 123: Tor Secure Access Package for USAGM [First Phase]Silvio RhattoSilvio Rhatto2023-04-04https://gitlab.torproject.org/tpo/onion-services/onionspray-log-parser/-/issues/7Add a flags on eotk-get-logs-from-s3 to select from/to dates2023-06-05T16:09:03ZSilvio RhattoAdd a flags on eotk-get-logs-from-s3 to select from/to dates* [x] Add a flags on `eotk-get-logs-from-s3` to allowing filtering logs by a data range or a single month. Only logs in that range (or in that month) should be copied.
* [x] Inform S123 analytics when this flag is ready to be tested.* [x] Add a flags on `eotk-get-logs-from-s3` to allowing filtering logs by a data range or a single month. Only logs in that range (or in that month) should be copied.
* [x] Inform S123 analytics when this flag is ready to be tested.Silvio RhattoSilvio Rhatto2023-05-31https://gitlab.torproject.org/tpo/tpa/team/-/issues/41258materculae out of disk space2023-09-21T01:51:41ZKezmaterculae out of disk spaceprevious ticket: #40826
it's been a year, and nagios is complaining about materculae's /srv partition
```
# df -h /srv
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_materculae-srv 147G 135G 4.3G 97%...previous ticket: #40826
it's been a year, and nagios is complaining about materculae's /srv partition
```
# df -h /srv
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_materculae-srv 147G 135G 4.3G 97% /srv
```
in the previous ticket (#40826) @anarcat changed the warning threshold, which is why this warning popped up now.
according to grafana, the usage has only been about 15G in the past year, and the growth is linear. we could add another 20G and revisit in a year, or throw 40G or 60G at it to push things further down the road.
![image](/uploads/e8ddf8b69703273f73d891586f7fc137/image.png)anarcatanarcat2023-09-22https://gitlab.torproject.org/tpo/onion-services/onionspray/-/issues/35MetricsPort support2024-02-01T05:18:15ZSilvio RhattoMetricsPort support# Tasks
* [x] Add `MetricsPort` and `MetricsPortPolicy` support.
* [x] Document how to monitor Onion Services.
# Time estimation
* Complexity: very small (0.5 day)
* Uncertainty: low (x1.1)
* [Reference](https://jacobian.org/2021/may/...# Tasks
* [x] Add `MetricsPort` and `MetricsPortPolicy` support.
* [x] Document how to monitor Onion Services.
# Time estimation
* Complexity: very small (0.5 day)
* Uncertainty: low (x1.1)
* [Reference](https://jacobian.org/2021/may/25/my-estimation-technique/) (adapted)Onionspray 1.6.0Silvio RhattoSilvio Rhatto2024-01-31https://gitlab.torproject.org/tpo/tpa/team/-/issues/41372pg backups filling up on bungei2024-03-26T15:15:15Zanarcatpg backups filling up on bungeisimilar to #41361 except now it's the `/srv/backups/pg` partition that's filling up...
1 year graph:
![image](/uploads/6500ce9736e25737fd16357e8d1f0d19/image.png)
https://grafana.torproject.org/d/zbCoGRjnz/disk-usage?orgId=1&from=now-1...similar to #41361 except now it's the `/srv/backups/pg` partition that's filling up...
1 year graph:
![image](/uploads/6500ce9736e25737fd16357e8d1f0d19/image.png)
https://grafana.torproject.org/d/zbCoGRjnz/disk-usage?orgId=1&from=now-1y&to=now&var-class=All&var-instance=bungei.torproject.org
30 days:
![image](/uploads/8b193a1cc848d97cde37ab43b49d2c77/image.png)
https://grafana.torproject.org/d/zbCoGRjnz/disk-usage?orgId=1&from=now-30d&to=now&var-class=All&var-instance=bungei.torproject.org
change rate is -1TB per month according to grafana.
/cc @gkanarcatanarcat2024-03-21https://gitlab.torproject.org/tpo/onion-services/onionspray-log-parser/-/issues/10Output template2024-03-28T14:23:08ZSilvio RhattoOutput template# Tasks
* [ ] Support for output with custom templating.
* [ ] Support for Markdown table output.
# Time estimation
* Complexity: very small (0.5 day)
* Uncertainty: low (x1.1)
* [Reference](https://jacobian.org/2021/may/25/my-estimat...# Tasks
* [ ] Support for output with custom templating.
* [ ] Support for Markdown table output.
# Time estimation
* Complexity: very small (0.5 day)
* Uncertainty: low (x1.1)
* [Reference](https://jacobian.org/2021/may/25/my-estimation-technique/) (adapted)Silvio RhattoSilvio Rhatto2024-04-01https://gitlab.torproject.org/tpo/onion-services/onionprobe/-/issues/80Enhanced Grafana dashboard2024-03-27T21:45:05ZSilvio RhattoEnhanced Grafana dashboardEnhance the sample [exportable](https://grafana.com/docs/grafana/latest/dashboards/export-import/) Grafana Dashboard for Onion Services monitoring, including:
* [ ] Lists of expiring X.509 certificates (next days/weeks/month/quarter; cu...Enhance the sample [exportable](https://grafana.com/docs/grafana/latest/dashboards/export-import/) Grafana Dashboard for Onion Services monitoring, including:
* [ ] Lists of expiring X.509 certificates (next days/weeks/month/quarter; current quarter; etc).
* [ ] Enhanced metrics from tpo/onion-services/onionprobe#78.Onionprobe 1.2.0Silvio RhattoSilvio Rhatto2024-05-16https://gitlab.torproject.org/tpo/onion-services/onionprobe/-/issues/78Enhanced metrics for Onion Service descriptors2024-03-27T21:44:54ZSilvio RhattoEnhanced metrics for Onion Service descriptorsImplement additional metrics for Onion Service descriptors.
That need:
* A better way to parse descriptors would enable many other metrics.
* Some patches sent upstream to Stem.
Some fields that could get measurements:
* From the out...Implement additional metrics for Onion Service descriptors.
That need:
* A better way to parse descriptors would enable many other metrics.
* Some patches sent upstream to Stem.
Some fields that could get measurements:
* From the outer descriptor wrapper:
* [ ] "descriptor-lifetime".
* [ ] "revision-counter".
* From the first layer of encryption:
* [ ] "[caa-critical](https://gitlab.torproject.org/tpo/core/torspec/-/blob/main/proposals/343-rend-caa.txt)".
* From the second layer of encryption:
* [ ] "single-onion-service".
* [ ] "pow-params": an indirect way to measure DoS for PoW-enabled
services (by measuring the PoW settings in the descriptor),
which depends on tpo/core/tor#40634 to be implemented.
* [ ] "[caa](https://gitlab.torproject.org/tpo/core/torspec/-/blob/main/proposals/343-rend-caa.txt)".
Other measurements:
* [ ] Metrics for the descriptor and inner layer sizes.Onionprobe 1.2.0Silvio RhattoSilvio Rhatto2024-05-16https://gitlab.torproject.org/tpo/onion-services/onionspray-log-parser/-/issues/11Slowness on onionspray-get-logs-from-s3fs2024-03-28T13:25:06ZSilvio RhattoSlowness on onionspray-get-logs-from-s3fs# Tasks
* [ ] Investigate why [onionspray-get-logs-from-s3fs][] is being slow, and how that can be fixed.
* [ ] If can't be fixed easily, recomend users to try [onionspray-get-logs-from-s3][] first.
[onionspray-get-logs-from-s3fs]: htt...# Tasks
* [ ] Investigate why [onionspray-get-logs-from-s3fs][] is being slow, and how that can be fixed.
* [ ] If can't be fixed easily, recomend users to try [onionspray-get-logs-from-s3][] first.
[onionspray-get-logs-from-s3fs]: https://gitlab.torproject.org/tpo/onion-services/onionspray-log-parser/-/blob/main/onionspray-get-logs-from-s3fs
[onionspray-get-logs-from-s3]: https://gitlab.torproject.org/tpo/onion-services/onionspray-log-parser/-/blob/main/onionspray-get-logs-from-s3
# Time estimation
* Complexity: very small (0.5 day)
* Uncertainty: low (x1.1)
* [Reference](https://jacobian.org/2021/may/25/my-estimation-technique/) (adapted)Silvio RhattoSilvio Rhattohttps://gitlab.torproject.org/tpo/tpa/team/-/issues/41526Deploy onionperf files parser on metricsdb-012024-03-07T14:23:37ZHiroDeploy onionperf files parser on metricsdb-01We need to deploy https://gitlab.torproject.org/tpo/network-health/metrics/tor_fusion/ on metricsdb-01.
Basically this thing will run, download onionperf files from collector and parse them. This will just happen once a day around 1am UT...We need to deploy https://gitlab.torproject.org/tpo/network-health/metrics/tor_fusion/ on metricsdb-01.
Basically this thing will run, download onionperf files from collector and parse them. This will just happen once a day around 1am UTC as at midnight is when collector fetches the archives from the various onionperf clients.
It's a little rust app and was thinking to create a group and user like for the metrics-api. But maybe it's a bit overkill and I should just put it in the parser space?HiroHirohttps://gitlab.torproject.org/tpo/tpa/team/-/issues/41516metricsdb-01 root filesystem is full2024-02-05T20:09:05ZJérôme Charaouilavamind@torproject.orgmetricsdb-01 root filesystem is fullFor over a week, the root filesystem on `metricsdb-01` has been filled to 100%.
The cause seems to be related to logs lines such as this being added tens (even hundreds) of thousands of times every day:
Feb 05 04:05:37 metricsdb-01...For over a week, the root filesystem on `metricsdb-01` has been filled to 100%.
The cause seems to be related to logs lines such as this being added tens (even hundreds) of thousands of times every day:
Feb 05 04:05:37 metricsdb-01 run[3664186]: 2024-02-05 04:05:37,453 WARN o.t.m.d.p.WebStatsParser:114 ERROR: duplicate key value violates unique constraint "log_line_pkey"
Feb 05 04:05:37 metricsdb-01 run[3664186]: Detail: Key (digest)=(g4tX2M7Beig0hqfn2OaUHKGTpXTjel+p8wrfWoTzK+8) already exists.HiroHiro