sponsor-61-sims merge requestshttps://gitlab.torproject.org/jnewsome/sponsor-61-sims/-/merge_requests2021-12-17T16:11:08Zhttps://gitlab.torproject.org/jnewsome/sponsor-61-sims/-/merge_requests/19Move simulation logic to an external python script2021-12-17T16:11:08ZJim NewsomeMove simulation logic to an external python scriptThis MR migrates the body of the simulation logic into an external python script. For this MR I tried to minimize behavioral changes and cleanup. There's a lot more that could be done in terms of using native python instead of shelling o...This MR migrates the body of the simulation logic into an external python script. For this MR I tried to minimize behavioral changes and cleanup. There's a lot more that could be done in terms of using native python instead of shelling out for everything, moving things into subroutines etc., but I tried to keep this to the bare minimum to set us up so that future logic can be directly written in python (without having to try to encode it in yml and shell or calling out to over-specialized helper scripts with awkward interfaces)
I did end up rewriting the locale splitting logic in Python, because I had trouble getting the shell escaping right, and that piece is particularly better handled in Python.https://gitlab.torproject.org/jnewsome/sponsor-61-sims/-/merge_requests/17New baseline configuration2021-12-15T06:22:37ZJim NewsomeNew baseline configuration* Bump tornettools to get "0 bandwidth" fix
* Apply consensus parameters from 2021-12-10
* Use network data from when network flooding was happening for more accurate numbers.* Bump tornettools to get "0 bandwidth" fix
* Apply consensus parameters from 2021-12-10
* Use network data from when network flooding was happening for more accurate numbers.https://gitlab.torproject.org/jnewsome/sponsor-61-sims/-/merge_requests/16Use tor live network data only for a portion of a month2021-12-15T02:59:12ZHiroUse tor live network data only for a portion of a monthAs discussed in https://gitlab.torproject.org/tpo/network-health/metrics/analysis/-/issues/40008#note_2764299 this delete days were aren't interested in from the tornettool staging step.
If we wanted to add more days from october we cou...As discussed in https://gitlab.torproject.org/tpo/network-health/metrics/analysis/-/issues/40008#note_2764299 this delete days were aren't interested in from the tornettool staging step.
If we wanted to add more days from october we could do that too modifying the bash scripting of how we download archives from collector.
This MR is for main but maybe we just want to do a few tests on the data before considering merging this branch.
cc\ @mikeperryHiroHirohttps://gitlab.torproject.org/jnewsome/sponsor-61-sims/-/merge_requests/15Retry on gitlab system failures and timeouts2021-12-02T20:46:37ZJim NewsomeRetry on gitlab system failures and timeoutsAddresses https://gitlab.torproject.org/tpo/tpa/team/-/issues/40438Addresses https://gitlab.torproject.org/tpo/tpa/team/-/issues/40438https://gitlab.torproject.org/jnewsome/sponsor-61-sims/-/merge_requests/14Log monitoring2021-12-01T22:13:58ZJim NewsomeLog monitoringSome tweaks based on debugging https://gitlab.torproject.org/jnewsome/sponsor-61-sims/-/jobs/63872
* Save resource monitoring to a log instead of dumping to the job's stdout, and don't tail logs on stdout. Now that we have a web shell o...Some tweaks based on debugging https://gitlab.torproject.org/jnewsome/sponsor-61-sims/-/jobs/63872
* Save resource monitoring to a log instead of dumping to the job's stdout, and don't tail logs on stdout. Now that we have a web shell on the runner, we can still check these while the job is running if needed. This should make it easier to follow the rest of the job log.
* In the after_script, don't fail if the source `env` file is missing. This wasn't the root cause of the failure in job 63872, but in general we should be defensive in the after_script so that we don't inadvertently skip other post-processing due to inconsequential failures.https://gitlab.torproject.org/jnewsome/sponsor-61-sims/-/merge_requests/13Save tor warnings and errors in artifacts2021-12-01T21:03:00ZJim NewsomeSave tor warnings and errors in artifactshttps://gitlab.torproject.org/jnewsome/sponsor-61-sims/-/merge_requests/12Sendme_inc params2021-11-30T21:03:48ZMike PerrySendme_inc paramshttps://gitlab.torproject.org/jnewsome/sponsor-61-sims/-/merge_requests/11Set lenient group permissions on /shadow data2021-11-23T17:35:21ZJim NewsomeSet lenient group permissions on /shadow dataAddresses:
https://gitlab.torproject.org/tpo/tpa/team/-/issues/40476#note_2762378Addresses:
https://gitlab.torproject.org/tpo/tpa/team/-/issues/40476#note_2762378https://gitlab.torproject.org/jnewsome/sponsor-61-sims/-/merge_requests/10Use filter rules in rsync instead of naming individual sources2021-11-23T17:33:18ZJim NewsomeUse filter rules in rsync instead of naming individual sourcesThis gives some additional flexibility. e.g. we can use unrooted
and `**` patterns. It also ensures the directory structure stays
consistent.
Test run: https://gitlab.torproject.org/jnewsome/sponsor-61-sims/-/pipelines/19234This gives some additional flexibility. e.g. we can use unrooted
and `**` patterns. It also ensures the directory structure stays
consistent.
Test run: https://gitlab.torproject.org/jnewsome/sponsor-61-sims/-/pipelines/19234https://gitlab.torproject.org/jnewsome/sponsor-61-sims/-/merge_requests/9Split perfclients per country code.2021-11-22T19:20:47ZHiroSplit perfclients per country code.Split per country code also in live networkSplit per country code also in live networkhttps://gitlab.torproject.org/jnewsome/sponsor-61-sims/-/merge_requests/8Split perfclients per country code.2021-11-19T19:01:47ZHiroSplit perfclients per country code.Plot in onionperf and tornettools ALL clients and clients in DE and HK.
@jnewsome I might need some help checking the paths in the simulation. Is there a way I can have a test run that lasts no more then five minutes?
I guess I should tw...Plot in onionperf and tornettools ALL clients and clients in DE and HK.
@jnewsome I might need some help checking the paths in the simulation. Is there a way I can have a test run that lasts no more then five minutes?
I guess I should tweak:
```
PL_SIM_SCALE: '0.001' # increase to 0.1 maybe?
# Network is bootstrapped at 5m. Simulate 1m of traffic afterwards.
PL_SIM_TIME: '6m'
# Treat as converged after bootstrap, so that there's some data present
# for post-processing.
PL_SIM_CONVERGENCE_TIME_S: '300'
```Jim NewsomeJim Newsomehttps://gitlab.torproject.org/jnewsome/sponsor-61-sims/-/merge_requests/7Use /shadow volume for full results2021-11-02T01:51:34ZJim NewsomeUse /shadow volume for full resultshttps://gitlab.torproject.org/jnewsome/sponsor-61-sims/-/merge_requests/6Configure hidden services as non-anonymous2021-10-30T17:31:46ZJim NewsomeConfigure hidden services as non-anonymousHopefully this will speed them up a bit.Hopefully this will speed them up a bit.https://gitlab.torproject.org/jnewsome/sponsor-61-sims/-/merge_requests/5Storage and monitoring improvements2021-10-30T14:47:34ZJim NewsomeStorage and monitoring improvementsMonitoring:
* Dump pipeline config vars to stdout, so that we can see manual overrides
* Serve simdir over an onion service, so that we can inspect files in a running sim
* (Temporarily) also serve cache dir over an onion service, so th...Monitoring:
* Dump pipeline config vars to stdout, so that we can see manual overrides
* Serve simdir over an onion service, so that we can inspect files in a running sim
* (Temporarily) also serve cache dir over an onion service, so that we can download caches from previous runs
Save tornettools logs, since they hold some metadata about the sim. *Probably* redundant, but they're small.
In response to recent full sim failing due to too-large artifacts:
* Always save simdir cache, even on job failure, for example due to too-large-artifacts.
* Drop Non-perfclient tgen stdout fileshttps://gitlab.torproject.org/jnewsome/sponsor-61-sims/-/merge_requests/3Tweak artifacts2021-10-29T22:05:43ZJim NewsomeTweak artifacts* Save all stderr and exitcode files to check for errors
* Save all tgen and oniontrace stdout's
* Don't save full output for perfclients* Save all stderr and exitcode files to check for errors
* Save all tgen and oniontrace stdout's
* Don't save full output for perfclientshttps://gitlab.torproject.org/jnewsome/sponsor-61-sims/-/merge_requests/2Improve pipeline variable handling2021-10-28T19:59:28ZJim NewsomeImprove pipeline variable handlingAdd descriptions to variables, so that they show up pre-filled in the gitlab UI when manually triggering a pipeline.
Add "presets" for managing sets of overrides. e.g. the default preset is "test", which runs a very short simulation and...Add descriptions to variables, so that they show up pre-filled in the gitlab UI when manually triggering a pipeline.
Add "presets" for managing sets of overrides. e.g. the default preset is "test", which runs a very short simulation and doesn't archive the results. Currently the only other preset is "default", which doesn't override anything, running a large-ish simulationhttps://gitlab.torproject.org/jnewsome/sponsor-61-sims/-/merge_requests/1Mikeperry cc+logscrape squashed2021-10-22T16:36:24ZMike PerryMikeperry cc+logscrape squashed