builds: improve benchmarking
spun off from #32759 ... so (windows) annecdata benchmarking indicates several things
speedometer
- TB/MB are consistently operating at around 2/3rds of ESR
- uBO affects perf at ~5%
- which is not an issue and is a symptom more about how the test is run than real-world? IDK
- uBO will be in TB at some stage, and the benefits (less network fetches etc) outweigh any cons IMO
- @pierov mentioned reproducible builds, PGO, LTO and I mentioned mingw - cc @fabrizio
- I will update jetstream and motionmark data after I test them
- needs labels and/or moving to appropriate repo cc @richard
jetstream
- about an 8% hit: is this primarily due to API/pref flips?
- NI (well, I have ideas, but not important) why gecko "regressed": TB10 vs ESR78 results are legit
motionmark (when I add them)
- sizes
-
1000x1000= small screen |1400x900and1400x1000= medium screen - will use medium screen across all tests
- since RFP is disabled
- IDK how that affects realworld results (e.g. non 60FPS hardware)
-
- results seems to fluxuate a lot between releases
- also IDK if this test is relevant: e.g. the perf hit in FF118 looks huge, but then read 1848050#c2
thorin's hardware
- OS: Windows 11 Pro 64-bit
- CPU: AMD K19 (x12)
- RAM: 32.0GB Dual-Channel DDR4 @ 1598MHz
- Motherboard: Gigabyte Technology Co. Ltd. B550M DS3H AC (AM4)
- Graphics: 4095MB NVIDIA GeForce RTX 3060 (ASUStek Computer Inc)
methods/notes
- https://browserbench.org/
- no extensions (e.g. uBO impacts at least speedometer by ~5+%)
- no RFP (to get accurate perf)
- new (+ sanitized) browser session for each test (S, J, M)
- speedometer: two runs each test (using
test again, i.e in same tab), took least ± (standard deviation?) - LW = librewolf, WF = waterfox, AF = arkenfox
- All FF's except Beta and Nightly are portable versions (portable versions use mozilla's binaries)
- EDIT: redid FF tests, using PB windows, vs TB/MB
TB/MB results
PB windows
S (2.1) J (2.1) M (1.2) medium screen
FF115 239 ± 4.5 156 953
ESR115mingw 184 ± 2.0 150
TB13.0a 155 ± 1.9 144 939
--- ---
TB 65% 92%
ESR102.14 181 ± 3.4 135 874
TB12.5.2 114 ± 1.3 114 605 (418 with RFP)
MB12.5.1 115 ± 1.9 117
MB12.5.1+uBO 106 ± 3.1 114
--- ---
TB/MB 63% 85%
MB+uBO 58% 85%
ESR91.13 158 ± 4.4 106
TB11.5.8 104 ± 2.4 98
--- ---
TB 66% 92%
ESR78.15 128 ± 3.5 124
TB10.5.10 91 ± 1.1 114
--- ---
TB 71% 92%FF/fork/other results
non-PB windows
S (2.1) J (2.1) M (1.2) medium screen
Chrome 115 291 ± 5.4 239 1795
Edge 115 287 ± 3.2 237 1576
Opera 114 247 ± 3.7 230 1573
FF118 (N) 267 ± 3.9 175 1203 (bugzilla 1848050#c2)
FF117 (D) 272 ± 1.2 161 921
FF117 (B) 266 ± 4.7 163 990
FF116 244 ± 3.6 157 925
FF116 (AF) 239 ± 4.0 158 936
FF116 (LW) 181 ± 2.5 139 853 (762 with RFP)
--- ---
LW 74% 89%
FF115 240 ± 2.3 154 953
Mercury115 234 ± 1.8 152 1002
FF114 238 ± 3.0 152 901
FF113 218 ± 3.0 149 921
FF112 211 ± 2.5 148 1007
FF111 199 ± 2.0 141 1041
FF110 197 ± 2.3 140 984
FF109 194 ± 2.3 142 979
FF108 189 ± 2.1 137 906
FF107 183 ± 2.8 124 752
FF106 176 ± 2.2 124 770
FF105 181 ± 4.1 133 916
FF104 178 ± 5.1 121 621
FF103 179 ± 4.6 120 577
FF102 179 ± 2.9 129 761
FF102 (WF) 143 ± 2.9 123 566
--- ---
WF 80% n/a
---
OMG... LMFAO
WF Classic 90 ± 1.5 93 460
Basilisk 43 ± 1.2 66 396
Palemoon 37 ± 1.2 62 380Edited by Thorin