Skip to content
Snippets Groups Projects
  1. May 02, 2024
    • Pierre-Yves David's avatar
      Bug 1894617: ignore the root .vscode directory too; r=sheehan · 7bb19b13
      Pierre-Yves David authored
      It looks like I misunderstood what the original rules meant and all content of
      all `.vscode` directory should be ignored, but for the `.vscode/extensions.json`
      and `.vscode/tasks.json` files. Since these file are already tracked, they don't
      need a dedicated ignore rules. However other files does (e.g
      `.vscode/settings.json`, `.vscode/launch.json`, etc).
      
      So we remove the exception for the root `.vscode` directory.
      
      This is a follow up to 4952395ba0ec.
      
      Differential Revision: https://phabricator.services.mozilla.com/D209260
      7bb19b13
  2. May 01, 2024
    • Pierre-Yves David's avatar
      Bug 1894160: hgignore: simplify the egginfo pattern; r=sheehan · 3d9d459e
      Pierre-Yves David authored
      This lookahead expression prevents the use of more modern and efficient regexp
      engine. This slows down "hg status" and other operations.
      
      Since the exception are only about vendored content whose addition is managed by
      a script (`match vendor`), that script can deal with this exception by itself,
      and it does since the last changeset..
      
      So we drop the exception to unlock various performance improvements for status.
      
      ### Why does this improves things?
      
      There improvement can come from different sources:
      
      * Using the "re2" regexp engine to match ignored files and directories provide
        a performance boost for vanillia mercurial installation and fs-monitor one in
        various cases. To benefit from it, just install the "google-re2" packages and
        mercurial will automatically uses it.
      
      * Installing a Mercurial compiled with the Rust extensions unlock the use of a
        more efficient code path for status that performs the necessary action in a
        smarter and parallel ways, providing a significant boost. These extensions
        are available on Linux and MacOs and some distribution have started to enable
        them by default.
      
      * Moving to a more modern "dirstate" format. The dirstate tracks the state of
        the working copy. For a couple of years, Mercurial has a new format for this
        information that is more efficient to read and update and tracks finer
        grained information. This allow substantial improvement in the way we run
        status. The Rust extensions are required to efficiently using this format.
      
      * Using a pure-rust executable. Mercurial has a pure rust version (called
        "rhg") that can handled a limited set of commands. It run without the
        overhead of starting and initializing Python providing another very
        significant boost to performance… but obviously requiring the Rust code path
        to be usable.
      
      ### Quick Conclusion of the Benchmarks
      
      (Putting that first for people who just want a quick read.)
      
      * fsmonitor struggle on working copy with many modication,
      
      * Using the "re2" binding from "google-re2" helps, especially for these cases
      
      * On typical mozilla developer machine, the Rust variants match the fsmonitor
        performance at worse and exceed it in multiple cases. Especially it does not
        stuggle with the "many modification" case.
      
      * On smaller machine, the Rust variants still provide a solid and reliable
        performance win accross all operation. That make them preferable to fsmonitor.
      
      * The rust variants matches "git status" performance on equivalement workload.
        The pure Rust version significantly outperforms it.
      
      
      ### Benchmarks descriptions
      
      Machines
      --------
      
      We ran benchmark on two different machines:
      
      * A i7-7700K 4 physical / 4 logical cores released in Jan 2017
      
        To see performance in "low" parallelism case.
      
      * A i9-9900K 8 physical / 16 logical cores released in October 2019
      
        To see performance in a "high" parallism case.
      
      In both cases the repositories lived in a btrfs file system backed by solid
      state disks (ssd or nvme) and the machines had enough ram to keep caches in
      memory.
      
      I also ran benchmarks on a more modern i7-1370P release on Jan 2023, and the
      results were consistent with the i9-9900K ones.
      
      Variants
      --------
      
      Benchmarks were run with multiple variants of Mercurial:
      
        * python-re:
          * no Rust extensions used,
          * regex engine is the std-lib "re" module.
          * fsmonitor is disabled
          * using the dirstate-v1 format
        * python-re2:
          * no Rust extensions used,
          * regex engine is the std-lib "re" module.
          * fsmonitor is disabled
          * using the dirstate-v1 format
        * fsmonitor-re:
          * no Rust extensions used,
          * regex engine is the std-lib "re" module.
          * fsmonitor is enabled and working at its best
          * using the dirstate-v1 format
        * fsmonitor-re2:
          * no Rust extensions used,
          * regex engine is the std-lib "re" module.
          * fsmonitor is enabled and working at its best
          * using the dirstate-v1 format
        * rust-ds1:
          * Rust extensions are used,
          * regex engine from the Rust "regexp" crate.
          * fsmonitor is disabled
          * using the dirstate-v1 format
        * rust-ds2:
          * Rust extensions are used,
          * regex engine from the Rust "regexp" crate.
          * fsmonitor is disabled
          * using the dirstate-v2 format
        * rgh-ds1:
          * Pure rust executable is used,
          * regex engine from the Rust "regexp" crate.
          * fsmonitor is disabled
          * using the dirstate-v1 format
        * rgh-ds2:
          * Pure rust executable is used,
          * regex engine from the Rust "regexp" crate.
          * fsmonitor is disabled
          * using the dirstate-v2 format
      
      Commands
      --------
      
      We ran two kind of operations:
      
      * `hg status` with the default output.
          This command need to search for ignored and unknown files.
          In this case improving the regex engine usually provides significant performance gain.
      
      * `hg status --modified --added --removed --deleted`.
          This command only need to check the state of tracked files.
          In this case, improving the regex engine does not have much effect, but it
          is interesting to compare the performance of the various implementation.
      
      Working copies
      --------------
      
      Case 1: pristine-928b0540e421
      
          Working copy parent is 928b0540e421
            * 341 759 tracked files
            *  21 253 directories
            * no untracked files
      
      Case 2: pristine-8f96f8c756ae
      
          Working copy parent is 8f96f8c756ae
              (an older changeset I had dirty working copy for)
            * 246 855 tracked files
            *  15 047 directory
            * no untracked files
      
      Case 3: clean-8f96f8c756ae
      
          Working copy parent is 8f96f8c756ae
            * 246 855 tracked files
            *  23 540 directories
            *  79 901 ignored files
      
      Case 4: dirty-8f96f8c756ae
      
          Working copy parent is 8f96f8c756ae
            * 246 855 tracked files
            *  33 720 directories
            * 244 386   clean files
            *   1 065 modified files
            *     247   added files
            *   1 040 removed
            *     364 missing files
            *  63 455 unknown files
            *  79 915 ignored files
      
      ### Results Analysis
      
      (full, raw number after this section)
      
      About fsmonitor
      ---------------
      
      Before diving into the improvements related to regex engine, we can note that
      the benchmark show that fsmonitor provides a good boost in the pristine/clean cases, and
      a noticeable but disappointing improvement in the very dirty case.
      
                                 python-re fsmonitor-re
          pristine-928b0540e421:     1.884 →      0.293 (-85%)
          dirty-8f96f8c756ae:        2.157 →      1.440 (-33%)
      
      Surprisingly when only listing tracked file (during commit for example), fsmonitor actually
      get counter productive in the very dirty case
      
          pristine-928b0540e421:     1.313 →      0.297 (-77%)
          dirty-8f96f8c756ae:        0.993 →      1.272 (+28%)
      
      In addition to being disappointing in the the very dirty case. The performance
      with fsmonitor collapses when fsmonitor cannot use its cache. I observed 4
      seconds execution time while setting up the brenchmark..
      
      Improvement without involving Rust:
      -----------------------------------
      
      Using the re2 binding from the google-re2 package provides a small improvement
      to plain python execution (about 15%). This case is relevant because this is
      the one that will be used when fsmonitor cannot help or start.
      
                                 python-re  python-re2
          pristine-928b0540e421:      1.884 →   1.650 (-15%)
          dirty-8f96f8c756ae:         2.157 →   1.718 (-20%)
      
      It does not make a difference when only listing tracked files as the hgignore is not involved.
      
                                 python-re  python-re2
          pristine-928b0540e421:      1.313 →    1.332
          dirty-8f96f8c756ae:         0.993 →    0.998
      
      However, surprisingly, it helps fsmonitor quite a lot in in the dirty case
      (dirty-8f96f8c756ae). Bringing fsmonitor performance in line with the plain
      python one.
      
                         fsmonitor-re fsmonitor-re2
          list-unknown          1.440 →       1.012 (-30%)
          tracked only          1.272 →       0.840 (-34%)
      
      So to conclude being able to use the "re2" regex engine save up to ⅓ of the
      runtime of some operation and never slow things down. So that's a good win.
      
      
      Improvement involving Rust variants:
      ------------------------------------
      
      For the pristine-928b0540e421 case (all tracked files clean, no ignored files),
      Rust provides speed boost "equivalent" (or better) to the one from fsmonitor.
      The precise comparison depends of the parallelism level.
      
      With the 4 physical / 4 logical core machine. The Python+Rust version is slower
      than fsmonitor, using dirstate-v2 helping to close some of the gap with
      fsmonitor.  Using dirstate-v2 also allow the "rhg" version to become twice
      faster than the fsmonitor version. Also keep in mind that even when a bit
      slower, the performance of the rust version will be much more stable than
      fsmonitor.
      
          python-re2:    1.650
          fsmonitor-re2: 0.296 (-82%)
          rust-ds1:      0.542 (-67%)
          rust-ds2:      0.368 (-77%)
          rhg-ds1:       0.401 (-75%)
          rhg-ds2:       0.132 (-92%)
      
      With the 8 physical / 16 physical code machine, the Rust catch up with
      fsmonitor performance much quicker. The dirstate-v1 is a little slower, but the
      dirstate-v2 version is already faster. The pure rust is always faster.
      
          python-re2:    1.430
          fsmonitor-re2: 0.278 (-80%)
          rust-ds1:      0.359 (-74%)
          rust-ds2:      0.259 (-81%)
          rhg-ds1:       0.235 (-83%)
          rhg-ds2:       0.052 (-96%)
      
      
      Talking about parallism. We see that the code scale well, doubling the
      number of core bring about twice the performance which is great.
      
      
          pristine-928b0540e421     4/4    8/16
              rhg-ds1:            0.401 → 0.235 (× 1.70)
              rhg-ds2:            0.132 → 0.052 (× 2.54)
          clean-8f96f8c756ae
              rhg-ds1:            0.286 → 0.169 (× 1.70)
              rhg-ds2:            0.101 → 0.040 (× 2.52)
          dirty-8f96f8c756ae
              rhg-ds1:            0.380 → 0.234 (x 1.62)
              rhg-ds2:            0.232 → 0.124 (x 1.87)
      
      
      Comparing with git performance on the pristine-928b0540e421 case also yield
      great results. Surprisingly, the variant with a Python overhead still beat (or
      match) git performance in this case. The pure Rust executable is always
      significantly faster. Below is a comparison grouped by comparable formats.
      
          git status -s: 0.554 (without untracked cache)
          rust-ds1:      0.359 (- 35%)
          rhg-ds1:       0.235 (- 57%)
      
          git status -s: 0.232 (with untracked cache)
          rust-ds2:      0.259 (+ 11%)
          rhg-ds2:       0.052 (- 77%)
      
      
      The clean-8f96f8c756ae case (all tracked clean, many ignored files) show result
      result similar to pristine-928b0540e421. "Low" parallism give good gains
      without fully matching the fs monitor performance. The High parallism provide
      similar performance. In both case we gain the benefit of more stable
      performances.
      
              (cores)          4/4           8/16
              python-re2:    1.282        | 1.119
              fsmonitor-re2: 0.243 (-81%) | 0.225 (-80%)
              rust-ds1:      0.416 (-68%) | 0.282 (-75%)
              rust-ds2:      0.303 (-76%) | 0.222 (-80%)
              rhg-ds1:       0.286 (-78%) | 0.169 (-85%)
              rhg-ds2:       0.101 (-92%) | 0.040 (-96%)
      
      Things change quite a lot in the dirty-8f96f8c756ae case, where fsmonitor
      struggled. The Rust variants still provides great speedup, significantly
      beating the fsmonitor variants for both machines. (comparing to fsmonitor-re
      this time)
      
              (cores)          4/4           8/16
              fsmonitor-re:  1.440        | 1.501
              fsmonitor-re2: 1.012 (-30%) | 1.051 (-30%)
              rust-ds1:      0.624 (-56%) | 0.519 (-65%)
              rust-ds2:      0.553 (-62%) | 0.483 (-68%)
              rhg-ds1:       0.380 (-73%) | 0.234 (-84%)
              rhg-ds2:       0.232 (-83%) | 0.124 (-91%)
      
      
      Things is confirmed in the "listing tracked only" version of dirty-8f96f8c756ae
      case were fs monitor was not really improving the situation compared to Python.
      
              (cores)          4/4           8/16
              python-re:     0.993        | 0.843076
              python-re2:    0.998        | 0.843324
              fsmonitor-re:  1.272 (+28%) | 1.291313 (+53%)
              fsmonitor-re2: 0.840 (-15%) | 0.844374
              rust-ds1:      0.364 (-63%) | 0.273305 (-68%)
              rust-ds2:      0.301 (-70%) | 0.233230 (-72%)
              rhg-ds1:       0.231 (-77%) | 0.153346 (-82%)
              rhg-ds2:       0.099 (-90%) | 0.039545 (-95%)
      
      ### Full benchmark numbers for `hg status`
      
      Here are the exhaustive number, all time in seconds.
      
      Case 1: pristine-928b0540e421
      
          (4/4 cores i7-7700K Jan 2017)
      
              python-re:     1.884
              python-re2:    1.650
              fsmonitor-re:  0.293 (more about 4 second when confused)
              fsmonitor-re2: 0.296
              rust-ds1:      0.542
              rust-ds2:      0.368
              rhg-ds1:       0.401
              rhg-ds2:       0.132
      
          (8/16 cores i9-9900K CPU October 2018)
      
              python-re:     1.674
              python-re2:    1.430
              fsmonitor-re:  0.272
              fsmonitor-re2: 0.278
              rust-ds1:      0.359
              rust-ds2:      0.259
              rhg-ds1:       0.235
              rhg-ds2:       0.052
      
              For reference, I also gathered timing for `git status` on this machine and repo
      
              git status -s: 0.554 (without untracked cache)
              git status -s: 0.232 (with untracked cache)
      
      Case 2: pristine-8f96f8c756ae
      
          (4/4 cores i7-7700K)
      
              python-re:     1.306
              python-re2:    1.227
              fsmonitor-re:  0.243
              fsmonitor-re2: 0.242
              rust-ds1:      0.416
              rust-ds2:      0.308
              rhg-ds1:       0.287
              rhg-ds2:       0.102
      
          (8/16 cores i9-9900K CPU)
      
              python-re:     1.131
              python-re2:    1.076
              fsmonitor-re:  0.222
              fsmonitor-re2: 0.222
              rust-ds1:      0.279
              rust-ds2:      0.222
              rhg-ds1:       0.168
              rhg-ds2:       0.038
      
      Case 3: clean-8f96f8c756ae
      
          (4/4 cores i7-7700K)
      
              python-re:     1.294
              python-re2:    1.282
              fsmonitor-re:  0.241
              fsmonitor-re2: 0.243
              rust-ds1:      0.416
              rust-ds2:      0.303
              rhg-ds1:       0.286
              rhg-ds2:       0.101
      
          (8/16 cores i9-9900K CPU)
      
              python-re:     1.170
              python-re2:    1.119
              fsmonitor-re:  0.224
              fsmonitor-re2: 0.225
              rust-ds1:      0.282
              rust-ds2:      0.222
              rhg-ds1:       0.169
              rhg-ds2:       0.040
      
      Case 4: dirty-8f96f8c756ae
      
          (4/4 cores i7-7700K)
      
              python-re:     2.157
              python-re2:    1.718
              fsmonitor-re:  1.440
              fsmonitor-re2: 1.012
              rust-ds1:      0.624
              rust-ds2:      0.553
              rhg-ds1:       0.380
              rhg-ds2:       0.232
      
          (8/16 cores i9-9900K CPU)
      
              python-re:     2.031
              python-re2:    1.560
              fsmonitor-re:  1.501
              fsmonitor-re2: 1.051
              rust-ds1:      0.519
              rust-ds2:      0.483
              rhg-ds1:       0.234
              rhg-ds2:       0.124
      
      ### Benchmark numbers for `hg status --modified --added --removed --deleted`
      
      With this invocation, status no longer need to list directory content (or use
      cache to skip that step). Status just need to check the known list of tracked
      files.
      
      Case 1: pristine-928b0540e421
      
          (4/4 cores i7-7700K CPU)
      
              python-re:     1.313
              python-re2:    1.332
              fsmonitor-re:  0.297
              fsmonitor-re2: 0.296
              rust-ds1:      0.455
              rust-ds2:      0.369
              rhg-ds1:       0.316
              rhg-ds2:       0.130
      
          (8/16 cores i9-9900K CPU)
      
              python-re:     1.129
              python-re2:    1.133
              fsmonitor-re:  0.273
              fsmonitor-re2: 0.271
              rust-ds1:      0.330
              rust-ds2:      0.244
              rhg-ds1:       0.207
              rhg-ds2:       0.050
      
              For reference, I also gathered timing for `git status` on this machine and repo
      
              git status -s --untracked-files=no: 0.110
      
      Case 2: pristine-8f96f8c756ae
      
          (4/4 cores i7-7700K)
      
              python-re:     0.993
              python-re2:    0.987
              fsmonitor-re:  0.241
              fsmonitor-re2: 0.243
              rust-ds1:      0.358
              rust-ds2:      0.307
              rhg-ds1:       0.228
              rhg-ds2:       0.100
      
          (8/16 cores i9-9900K CPU)
      
              python-re:     0.856
              python-re2:    0.839
              fsmonitor-re:  0.221
              fsmonitor-re2: 0.222
              rust-ds1:      0.262
              rust-ds2:      0.221
              rhg-ds1:       0.152
              rhg-ds2:       0.038
      
      Case 3: clean-8f96f8c756ae
      
          (4/4 cores i7-7700K)
      
              python-re:     0.973
              python-re2:    0.979
              fsmonitor-re:  0.242
              fsmonitor-re2: 0.242
              rust-ds1:      0.357
              rust-ds2:      0.304
              rhg-ds1:       0.224
              rhg-ds2:       0.098
      
          (8/16 cores i9-9900K CPU)
      
              python-re:     0.838
              python-re2:    0.837
              fsmonitor-re:  0.222
              fsmonitor-re2: 0.221
              rust-ds1:      0.263
              rust-ds2:      0.219
              rhg-ds1:       0.152
              rhg-ds2:       0.037
      
      
      Case 4: dirty-8f96f8c756ae
      
          (4/4 cores i7-7700K)
      
              python-re:     0.993
              python-re2:    0.998
              fsmonitor-re:  1.272
              fsmonitor-re2: 0.840
              rust-ds1:      0.364
              rust-ds2:      0.301
              rhg-ds1:       0.231
              rhg-ds2:       0.099
      
          (8/16 cores i9-9900K CPU)
      
              python-re:     0.843
              python-re2:    0.843
              fsmonitor-re:  1.291
              fsmonitor-re2: 0.844
              rust-ds1:      0.273
              rust-ds2:      0.233
              rhg-ds1:       0.153
              rhg-ds2:       0.040
      
      Differential Revision: https://phabricator.services.mozilla.com/D208966
      3d9d459e
    • Pierre-Yves David's avatar
      Bug 1894160: hgignore: drop the negative lookahead assertion around vscode; r=sheehan · debf431a
      Pierre-Yves David authored
      This lookahead prevents the use of more modern and efficient regexp engine
      slowing down status.
      
      In practice we only have two vscode directory tracked in Mercurial:
      * the root one, that see active development,
      * the one in "remote/test/puppeteer/" that was never touched since its addition.
      
      It is easy to not match the root in the hgignore, but harder for the other one.
      
      Please note that once a file is tracked by Mercurial, the fact it is ignored or
      not no longer matters, so in practice this will only affect "future" addition.
      
      However the history shows that this addition are extremely rare (one in over 15
      years) and that the only occurrence is some venturing, where the vscode file
      seems less important.
      
      So dropping this exception seems fine, the small inconvenience of having to
      manually add the file in an hypothetical future is negligible compared to
      concrete performance improvement of common operation to everyone.
      
      See the other changesets dropping the second lookahead patterns for performance
      number.
      
      Differential Revision: https://phabricator.services.mozilla.com/D208967
      debf431a
  3. Apr 19, 2024
  4. Apr 08, 2024
    • Greg Mierzwinski's avatar
      Bug 1855674 - Modify pdfpaint test to run more PDFs. r=aglavic,perftest-reviewers · 7948524f
      Greg Mierzwinski authored
      This patch modifies the pdfpaint test to run more pdfs that are found in the Mozilla pdf.js repository. The pdfpaint test is also moved to it's own suite due to the number of PDFs now being tested. These PDFs are pulled in locally from a toolchain task called talos-pdfs. The *ignore files are modified since the pdfpaint folder now contains a symbolic link to the local PDFs that should not be commit in-tree.
      
      To handle running the large number of PDFs, chunking is added to the test with the chunk size being 100 PDFs. Each chunk runs each of the 100 PDFs 5 times. A CLI option is also added for local runs so that users can select a specific pdfpaint PDF to test. An additional issue with the subtest/pdf file name parsing is also fixed for this to work.
      
      Differential Revision: https://phabricator.services.mozilla.com/D205824
      7948524f
  5. Mar 26, 2024
  6. Mar 19, 2024
  7. Mar 18, 2024
  8. Mar 15, 2024
  9. Mar 02, 2024
  10. Feb 22, 2024
  11. Dec 14, 2023
  12. Dec 13, 2023
  13. Nov 28, 2023
  14. Oct 25, 2023
  15. Jun 19, 2023
  16. Apr 23, 2023
  17. Apr 18, 2023
  18. Feb 28, 2023
  19. Feb 13, 2023
  20. Jan 23, 2023
    • Hanna Jones's avatar
      Bug 1799699 - expand storybook args table docs r=mstriemer,tgiles · eb9fd58f
      Hanna Jones authored
      This is still far from perfect given the limitations of the Storybook web components package, but I figured this was worth putting up since it's still an improvement over the current state of our args tables (I think).
      
      I'm mostly leaving the default generated `custom-elements-manifest.json` alone save for filtering some internal properties we don't want documented since they shouldn't really be accessed directly. If it seems too strange to just have the `aria-label` attr documented we could possibly remove `attributes` from the docs for now (this happens because it's the only attr where the name is different from the property name).
      
      Open to feedback/thoughts on if this is useful or too wonky for now given the weirdness around how Storybook creates naming collisions.
      
      Differential Revision: https://phabricator.services.mozilla.com/D162599
      eb9fd58f
  21. Sep 27, 2022
  22. Aug 31, 2022
  23. Jul 11, 2022
  24. May 25, 2022
  25. Apr 27, 2022
  26. Mar 11, 2022
  27. Jan 25, 2022
  28. Jan 12, 2022
  29. Nov 16, 2021
  30. Nov 02, 2021
  31. Oct 18, 2021
  32. Sep 28, 2021
  33. Sep 27, 2021
    • criss's avatar
      Backed out 10 changesets (bug 1712151, bug 1724279, bug 1730712, bug 1717051,... · f2dcba95
      criss authored
      Backed out 10 changesets (bug 1712151, bug 1724279, bug 1730712, bug 1717051, bug 1723031, bug 1731145) for causing failures on test_yaml.py
      Backed out changeset 7f64d538701b (bug 1723031)
      Backed out changeset 394152994966 (bug 1723031)
      Backed out changeset 9bfeb01bcc9a (bug 1723031)
      Backed out changeset 3d283616a57d (bug 1730712)
      Backed out changeset bc677b409650 (bug 1724279)
      Backed out changeset 784c94c2f528 (bug 1723031)
      Backed out changeset 6e1bde40e3b4 (bug 1723031)
      Backed out changeset 7adf7e2136a3 (bug 1712151)
      Backed out changeset 2aef162b9a1b (bug 1717051)
      Backed out changeset 9beeb6d3d95b (bug 1731145)
      f2dcba95
    • Mitchell Hentges's avatar
      Bug 1731145: Don't ignore vendored package `*.egg-info` directories r=ahal · 88d28c21
      Mitchell Hentges authored
      The `*.egg-info` directories are needed for the packages to show up as
      "distributions" to `pip` and other environment-checking logic.
      
      We know that `*.egg-info` directories are cross-platform because they
      exist in the globally-usable `tar.gz` releases of packages.
      
      Differential Revision: https://phabricator.services.mozilla.com/D125909
      88d28c21
  34. Jan 19, 2021
  35. Nov 27, 2020
Loading