sponsor-61-sims merge requestshttps://gitlab.torproject.org/jnewsome/sponsor-61-sims/-/merge_requests2022-01-08T02:27:25Zhttps://gitlab.torproject.org/jnewsome/sponsor-61-sims/-/merge_requests/23Clean up old sim data2022-01-08T02:27:25ZJim NewsomeClean up old sim datahttps://gitlab.torproject.org/jnewsome/sponsor-61-sims/-/merge_requests/22Append torrc2021-12-17T19:21:09ZJim NewsomeAppend torrchttps://gitlab.torproject.org/jnewsome/sponsor-61-sims/-/merge_requests/21Add option to override server bandwidth2021-12-17T18:35:17ZJim NewsomeAdd option to override server bandwidthhttps://gitlab.torproject.org/jnewsome/sponsor-61-sims/-/merge_requests/20Add option to override perfclient bandwidth2021-12-17T17:52:17ZJim NewsomeAdd option to override perfclient bandwidthhttps://gitlab.torproject.org/jnewsome/sponsor-61-sims/-/merge_requests/19Move simulation logic to an external python script2021-12-17T16:11:08ZJim NewsomeMove simulation logic to an external python scriptThis MR migrates the body of the simulation logic into an external python script. For this MR I tried to minimize behavioral changes and cleanup. There's a lot more that could be done in terms of using native python instead of shelling o...This MR migrates the body of the simulation logic into an external python script. For this MR I tried to minimize behavioral changes and cleanup. There's a lot more that could be done in terms of using native python instead of shelling out for everything, moving things into subroutines etc., but I tried to keep this to the bare minimum to set us up so that future logic can be directly written in python (without having to try to encode it in yml and shell or calling out to over-specialized helper scripts with awkward interfaces)
I did end up rewriting the locale splitting logic in Python, because I had trouble getting the shell escaping right, and that piece is particularly better handled in Python.https://gitlab.torproject.org/jnewsome/sponsor-61-sims/-/merge_requests/18Draft: Add options to override perfclient bandwidth2021-12-17T16:30:20ZJim NewsomeDraft: Add options to override perfclient bandwidthProgress on #8Progress on #8https://gitlab.torproject.org/jnewsome/sponsor-61-sims/-/merge_requests/17New baseline configuration2021-12-15T06:22:37ZJim NewsomeNew baseline configuration* Bump tornettools to get "0 bandwidth" fix
* Apply consensus parameters from 2021-12-10
* Use network data from when network flooding was happening for more accurate numbers.* Bump tornettools to get "0 bandwidth" fix
* Apply consensus parameters from 2021-12-10
* Use network data from when network flooding was happening for more accurate numbers.https://gitlab.torproject.org/jnewsome/sponsor-61-sims/-/merge_requests/16Use tor live network data only for a portion of a month2021-12-15T02:59:12ZHiroUse tor live network data only for a portion of a monthAs discussed in https://gitlab.torproject.org/tpo/network-health/metrics/analysis/-/issues/40008#note_2764299 this delete days were aren't interested in from the tornettool staging step.
If we wanted to add more days from october we cou...As discussed in https://gitlab.torproject.org/tpo/network-health/metrics/analysis/-/issues/40008#note_2764299 this delete days were aren't interested in from the tornettool staging step.
If we wanted to add more days from october we could do that too modifying the bash scripting of how we download archives from collector.
This MR is for main but maybe we just want to do a few tests on the data before considering merging this branch.
cc\ @mikeperryHiroHirohttps://gitlab.torproject.org/jnewsome/sponsor-61-sims/-/merge_requests/15Retry on gitlab system failures and timeouts2021-12-02T20:46:37ZJim NewsomeRetry on gitlab system failures and timeoutsAddresses https://gitlab.torproject.org/tpo/tpa/team/-/issues/40438Addresses https://gitlab.torproject.org/tpo/tpa/team/-/issues/40438https://gitlab.torproject.org/jnewsome/sponsor-61-sims/-/merge_requests/14Log monitoring2021-12-01T22:13:58ZJim NewsomeLog monitoringSome tweaks based on debugging https://gitlab.torproject.org/jnewsome/sponsor-61-sims/-/jobs/63872
* Save resource monitoring to a log instead of dumping to the job's stdout, and don't tail logs on stdout. Now that we have a web shell o...Some tweaks based on debugging https://gitlab.torproject.org/jnewsome/sponsor-61-sims/-/jobs/63872
* Save resource monitoring to a log instead of dumping to the job's stdout, and don't tail logs on stdout. Now that we have a web shell on the runner, we can still check these while the job is running if needed. This should make it easier to follow the rest of the job log.
* In the after_script, don't fail if the source `env` file is missing. This wasn't the root cause of the failure in job 63872, but in general we should be defensive in the after_script so that we don't inadvertently skip other post-processing due to inconsequential failures.https://gitlab.torproject.org/jnewsome/sponsor-61-sims/-/merge_requests/13Save tor warnings and errors in artifacts2021-12-01T21:03:00ZJim NewsomeSave tor warnings and errors in artifactshttps://gitlab.torproject.org/jnewsome/sponsor-61-sims/-/merge_requests/12Sendme_inc params2021-11-30T21:03:48ZMike PerrySendme_inc paramshttps://gitlab.torproject.org/jnewsome/sponsor-61-sims/-/merge_requests/11Set lenient group permissions on /shadow data2021-11-23T17:35:21ZJim NewsomeSet lenient group permissions on /shadow dataAddresses:
https://gitlab.torproject.org/tpo/tpa/team/-/issues/40476#note_2762378Addresses:
https://gitlab.torproject.org/tpo/tpa/team/-/issues/40476#note_2762378https://gitlab.torproject.org/jnewsome/sponsor-61-sims/-/merge_requests/10Use filter rules in rsync instead of naming individual sources2021-11-23T17:33:18ZJim NewsomeUse filter rules in rsync instead of naming individual sourcesThis gives some additional flexibility. e.g. we can use unrooted
and `**` patterns. It also ensures the directory structure stays
consistent.
Test run: https://gitlab.torproject.org/jnewsome/sponsor-61-sims/-/pipelines/19234This gives some additional flexibility. e.g. we can use unrooted
and `**` patterns. It also ensures the directory structure stays
consistent.
Test run: https://gitlab.torproject.org/jnewsome/sponsor-61-sims/-/pipelines/19234https://gitlab.torproject.org/jnewsome/sponsor-61-sims/-/merge_requests/9Split perfclients per country code.2021-11-22T19:20:47ZHiroSplit perfclients per country code.Split per country code also in live networkSplit per country code also in live networkhttps://gitlab.torproject.org/jnewsome/sponsor-61-sims/-/merge_requests/8Split perfclients per country code.2021-11-19T19:01:47ZHiroSplit perfclients per country code.Plot in onionperf and tornettools ALL clients and clients in DE and HK.
@jnewsome I might need some help checking the paths in the simulation. Is there a way I can have a test run that lasts no more then five minutes?
I guess I should tw...Plot in onionperf and tornettools ALL clients and clients in DE and HK.
@jnewsome I might need some help checking the paths in the simulation. Is there a way I can have a test run that lasts no more then five minutes?
I guess I should tweak:
```
PL_SIM_SCALE: '0.001' # increase to 0.1 maybe?
# Network is bootstrapped at 5m. Simulate 1m of traffic afterwards.
PL_SIM_TIME: '6m'
# Treat as converged after bootstrap, so that there's some data present
# for post-processing.
PL_SIM_CONVERGENCE_TIME_S: '300'
```Jim NewsomeJim Newsomehttps://gitlab.torproject.org/jnewsome/sponsor-61-sims/-/merge_requests/7Use /shadow volume for full results2021-11-02T01:51:34ZJim NewsomeUse /shadow volume for full resultshttps://gitlab.torproject.org/jnewsome/sponsor-61-sims/-/merge_requests/6Configure hidden services as non-anonymous2021-10-30T17:31:46ZJim NewsomeConfigure hidden services as non-anonymousHopefully this will speed them up a bit.Hopefully this will speed them up a bit.https://gitlab.torproject.org/jnewsome/sponsor-61-sims/-/merge_requests/5Storage and monitoring improvements2021-10-30T14:47:34ZJim NewsomeStorage and monitoring improvementsMonitoring:
* Dump pipeline config vars to stdout, so that we can see manual overrides
* Serve simdir over an onion service, so that we can inspect files in a running sim
* (Temporarily) also serve cache dir over an onion service, so th...Monitoring:
* Dump pipeline config vars to stdout, so that we can see manual overrides
* Serve simdir over an onion service, so that we can inspect files in a running sim
* (Temporarily) also serve cache dir over an onion service, so that we can download caches from previous runs
Save tornettools logs, since they hold some metadata about the sim. *Probably* redundant, but they're small.
In response to recent full sim failing due to too-large artifacts:
* Always save simdir cache, even on job failure, for example due to too-large-artifacts.
* Drop Non-perfclient tgen stdout fileshttps://gitlab.torproject.org/jnewsome/sponsor-61-sims/-/merge_requests/4Dump vars2021-10-30T14:47:34ZJim NewsomeDump vars