Skip to content
Snippets Groups Projects

ESR60-based Tor Browser bundles are not built reproducibly with Stylo enabled using rustc > 1.25.0

    • Closed (moved) Issue created by Georg Koppen @gk

    Contrary to our testing results the .dmg bundles we produce for 8.0a9 are differing on two machines. And it turns out Linux bundles are affected as well (see: #27937 (moved) for the original Linux bug report).

    To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information

    Child items 0

    • No child items are currently assigned. Use child items to break down this issue into smaller parts.

    Activity

    • Georg Koppen added GeorgKoppen201810 TorBrowserTeam201810R component::applications/tor browser owner::tbb-team priority::immediate resolution::fixed severity::normal status::closed tbb-backported tbb-rbm type::defect labels

      added GeorgKoppen201810 TorBrowserTeam201810R component::applications/tor browser owner::tbb-team priority::immediate resolution::fixed severity::normal status::closed tbb-backported tbb-rbm type::defect labels

    • Georg Koppen
      Georg Koppen @gk ·
      Author

      Only XUL is different in our case:

      --- /dev/fd/63	2018-06-24 08:54:28.175516977 +0200
+++ /dev/fd/62	2018-06-24 08:54:28.175516977 +0200
@@ -201,8 +201,8 @@
 00000c80: 0000 0000 0000 0000 0000 0000 0000 0000  ................
 00000c90: 0000 0000 0000 0000 104f 5805 a616 0000  .........OX.....
 00000ca0: 0000 0000 0000 0000 0000 0000 0000 0000  ................
-00000cb0: 1b00 0000 1800 0000 ddc1 d65d f3d8 3a65  ...........]..:e
-00000cc0: 8465 a64e d247 5b4d 2400 0000 1000 0000  .e.N.G[M$.......
+00000cb0: 1b00 0000 1800 0000 18a4 cc7a 95d7 325d  ...........z..2]
+00000cc0: 96c5 6f08 51ec 0b4e 2400 0000 1000 0000  ..o.Q..N$.......
 00000cd0: 0007 0a00 0006 0a00 0c00 0000 5800 0000  ............X...
 00000ce0: 1800 0000 0200 0000 0000 1600 0000 0100  ................
 00000cf0: 2f53 7973 7465 6d2f 4c69 6272 6172 792f  /System/Library/
@@ -3732284,59 +3732284,59 @@
 038f33b0: ff48 8b07 488b 4f08 4889 8d70 ffff ff48  .H..H.O.H..p...H
 038f33c0: 8985 68ff ffff 488d 051e 432a 0048 8985  ..h...H...C*.H..
 038f33d0: 38ff ffff 48c7 8540 ffff ff06 0000 0048  8...H..@.......H
-038f33e0: 8d3d 9adf bf01 ff17 8078 2100 0f85 da01  .=.......x!.....
+038f33e0: 8d3d 9adf bf01 ff17 8078 2100 0f85 d801  .=.......x!.....
 038f33f0: 0000 488d 3d87 dfbf 01ff 1780 7820 0075  ..H.=.......x .u
 038f3400: 1f48 8d3d 78df bf01 ff17 4889 c348 8d3d  .H.=x.....H..H.=
 038f3410: fc95 ffff 4889 dee8 c0c1 0100 c643 2001  ....H........C .
-038f3420: 488d 3d59 dfbf 01ff 1748 8338 0174 6748  H.=Y.....H.8.tgH
+038f3420: 488d 3d59 dfbf 01ff 1748 8338 0174 6548  H.=Y.....H.8.teH
 038f3430: 8d3d 4adf bf01 ff17 660f 6f40 100f 57c9  .=J.....f.o@..W.
-038f3440: 0f29 4810 b901 0000 0066 480f 6ec9 488b  .)H......fH.n.H.
-038f3450: 0866 0f7f 0848 85c9 7429 6648 0f7e c348  .f...H..t)fH.~.H
-038f3460: 85db 741f 660f 70c0 4e66 490f 7ec6 4889  ..t.f.p.NfI.~.H.
-038f3470: df41 ff16 4983 7e08 0074 0848 89df e8c7  .A..I.~..t.H....
-038f3480: b001 0048 8d3d f6de bf01 ff17 4883 3801  ...H.=......H.8.
-038f3490: 0f85 8102 0000 488d 3de3 debf 01ff 1748  ......H.=......H
-038f34a0: 8378 0800 7477 e895 f2ff ff48 89c3 4889  .x..tw.....H..H.
-038f34b0: 9d48 ffff ff48 8b45 9048 8945 d048 8b45  .H...H.E.H.E.H.E
-038f34c0: 8848 8945 c848 8b45 8048 8945 c048 8b85  .H.E.H.E.H.E.H..
-038f34d0: 78ff ffff 4889 45b8 488b 8568 ffff ff48  x...H.E.H..h...H
-038f34e0: 8b8d 70ff ffff 4889 4db0 4889 45a8 488d  ..p...H.M.H.E.H.
-038f34f0: 7dd8 488d b548 ffff ff48 8d55 a8e8 8ef9  }.H..H...H.U....
-038f3500: ffff f048 ff0b 0f85 a100 0000 488d bd48  ...H........H..H
-038f3510: ffff ffe8 58c0 ffff e990 0000 0048 8d3d  ....X........H.=
-038f3520: 5cde bf01 ff17 48c7 4008 ffff ffff 488d  \.....H.@.....H.
-038f3530: 4808 488d 5010 4889 9548 ffff ff48 898d  H.H.P.H..H...H..
-038f3540: 50ff ffff 488b 7010 4885 f60f 84b0 0100  P...H.p.H.......
-038f3550: 0048 8d3d 28de bf01 ff17 4889 c348 8b45  .H.=(.....H..H.E
-038f3560: 9048 8945 d048 8b43 1848 8b4d 8848 894d  .H.E.H.C.H.M.H.M
-038f3570: c848 8b4d 8048 894d c048 8b8d 78ff ffff  .H.M.H.M.H..x...
-038f3580: 4889 4db8 488b 8d68 ffff ff48 8b95 70ff  H.M.H..h...H..p.
-038f3590: ffff 4889 55b0 4889 4da8 488d 7dd8 488d  ..H.U.H.M.H.}.H.
-038f35a0: 55a8 ff50 3048 c743 0800 0000 008b 45d9  U..P0H.C......E.
-038f35b0: 8945 e88a 45d8 0fb7 4ddd 6689 4dec 8a4d  .E..E...M.f.M..M
-038f35c0: df88 4dee 3c04 0f85 8000 0000 e86f f1ff  ..M.<........o..
-038f35d0: ff48 89c3 4889 9d48 ffff ff48 8b45 9048  .H..H..H...H.E.H
-038f35e0: 8945 d048 8b45 8848 8945 c848 8b45 8048  .E.H.E.H.E.H.E.H
-038f35f0: 8945 c048 8b85 78ff ffff 4889 45b8 488b  .E.H..x...H.E.H.
-038f3600: 8568 ffff ff48 8b8d 70ff ffff 4889 4db0  .h...H..p...H.M.
-038f3610: 4889 45a8 488d 7d98 488d b548 ffff ff48  H.E.H.}.H..H...H
-038f3620: 8d55 a8e8 68f8 ffff f048 ff0b 750c 488d  .U..h....H..u.H.
-038f3630: bd48 ffff ffe8 36bf ffff 807d 9803 7531  .H....6....}..u1
-038f3640: 4881 c4c0 0000 005b 415e 5dc3 8845 9848  H......[A^]..E.H
-038f3650: 8b45 e08b 4de8 894d 990f b74d ec66 894d  .E..M..M...M.f.M
-038f3660: 9d8a 4dee 884d 9f48 8945 a080 7d98 0374  ..M..M.H.E..}..t
-038f3670: cf48 8b45 9848 8b4d a048 894d e048 8945  .H.E.H.M.H.M.H.E
-038f3680: d848 8d85 38ff ffff 4889 8548 ffff ff48  .H..8...H..H...H
-038f3690: 8d05 dab6 ffff 4889 8550 ffff ff48 8d45  ......H..P...H.E
-038f36a0: d848 8985 58ff ffff 488d 0501 99ff ff48  .H..X...H......H
-038f36b0: 8985 60ff ffff 488d 052b a7b6 0148 8945  ..`...H..+...H.E
-038f36c0: a848 c745 b002 0000 0048 8d05 f863 2a00  .H.E.....H...c*.
-038f36d0: 4889 45b8 48c7 45c0 0200 0000 488d 8548  H.E.H.E.....H..H
-038f36e0: ffff ff48 8945 c848 c745 d002 0000 0048  ...H.E.H.E.....H
-038f36f0: 8d35 12a7 b601 488d 7da8 e881 e5ff ff0f  .5....H.}.......
-038f3700: 0b48 8d3d 78dc bf01 ff17 48c7 4008 0000  .H.=x.....H.@...
-038f3710: 0000 e98f fdff ff48 8d3d eabb b601 e82d  .......H.=.....-
-038f3720: 8f00 000f 0b66 662e 0f1f 8400 0000 0000  .....ff.........
+038f3440: 0f29 4810 b901 0000 0066 480f 6ec9 4883  .)H......fH.n.H.
+038f3450: 3800 660f 7f08 7429 6648 0f7e c348 85db  8.f...t)fH.~.H..
+038f3460: 741f 660f 70c0 4e66 490f 7ec6 4889 df41  t.f.p.NfI.~.H..A
+038f3470: ff16 4983 7e08 0074 0848 89df e8c9 b001  ..I.~..t.H......
+038f3480: 0048 8d3d f8de bf01 ff17 4883 3801 0f85  .H.=......H.8...
+038f3490: 8102 0000 488d 3de5 debf 01ff 1748 8378  ....H.=......H.x
+038f34a0: 0800 7477 e897 f2ff ff48 89c3 4889 9d48  ..tw.....H..H..H
+038f34b0: ffff ff48 8b45 9048 8945 d048 8b45 8848  ...H.E.H.E.H.E.H
+038f34c0: 8945 c848 8b45 8048 8945 c048 8b85 78ff  .E.H.E.H.E.H..x.
+038f34d0: ffff 4889 45b8 488b 8568 ffff ff48 8b8d  ..H.E.H..h...H..
+038f34e0: 70ff ffff 4889 4db0 4889 45a8 488d 7dd8  p...H.M.H.E.H.}.
+038f34f0: 488d b548 ffff ff48 8d55 a8e8 90f9 ffff  H..H...H.U......
+038f3500: f048 ff0b 0f85 a100 0000 488d bd48 ffff  .H........H..H..
+038f3510: ffe8 5ac0 ffff e990 0000 0048 8d3d 5ede  ..Z........H.=^.
+038f3520: bf01 ff17 48c7 4008 ffff ffff 488d 4808  ....H.@.....H.H.
+038f3530: 488d 5010 4889 9548 ffff ff48 898d 50ff  H.P.H..H...H..P.
+038f3540: ffff 488b 7010 4885 f60f 84b0 0100 0048  ..H.p.H........H
+038f3550: 8d3d 2ade bf01 ff17 4889 c348 8b45 9048  .=*.....H..H.E.H
+038f3560: 8945 d048 8b43 1848 8b4d 8848 894d c848  .E.H.C.H.M.H.M.H
+038f3570: 8b4d 8048 894d c048 8b8d 78ff ffff 4889  .M.H.M.H..x...H.
+038f3580: 4db8 488b 8d68 ffff ff48 8b95 70ff ffff  M.H..h...H..p...
+038f3590: 4889 55b0 4889 4da8 488d 7dd8 488d 55a8  H.U.H.M.H.}.H.U.
+038f35a0: ff50 3048 c743 0800 0000 008b 45d9 8945  .P0H.C......E..E
+038f35b0: e88a 45d8 0fb7 4ddd 6689 4dec 8a4d df88  ..E...M.f.M..M..
+038f35c0: 4dee 3c04 0f85 8000 0000 e871 f1ff ff48  M.<........q...H
+038f35d0: 89c3 4889 9d48 ffff ff48 8b45 9048 8945  ..H..H...H.E.H.E
+038f35e0: d048 8b45 8848 8945 c848 8b45 8048 8945  .H.E.H.E.H.E.H.E
+038f35f0: c048 8b85 78ff ffff 4889 45b8 488b 8568  .H..x...H.E.H..h
+038f3600: ffff ff48 8b8d 70ff ffff 4889 4db0 4889  ...H..p...H.M.H.
+038f3610: 45a8 488d 7d98 488d b548 ffff ff48 8d55  E.H.}.H..H...H.U
+038f3620: a8e8 6af8 ffff f048 ff0b 750c 488d bd48  ..j....H..u.H..H
+038f3630: ffff ffe8 38bf ffff 807d 9803 7531 4881  ....8....}..u1H.
+038f3640: c4c0 0000 005b 415e 5dc3 8845 9848 8b45  .....[A^]..E.H.E
+038f3650: e08b 4de8 894d 990f b74d ec66 894d 9d8a  ..M..M...M.f.M..
+038f3660: 4dee 884d 9f48 8945 a080 7d98 0374 cf48  M..M.H.E..}..t.H
+038f3670: 8b45 9848 8b4d a048 894d e048 8945 d848  .E.H.M.H.M.H.E.H
+038f3680: 8d85 38ff ffff 4889 8548 ffff ff48 8d05  ..8...H..H...H..
+038f3690: dcb6 ffff 4889 8550 ffff ff48 8d45 d848  ....H..P...H.E.H
+038f36a0: 8985 58ff ffff 488d 0503 99ff ff48 8985  ..X...H......H..
+038f36b0: 60ff ffff 488d 052d a7b6 0148 8945 a848  `...H..-...H.E.H
+038f36c0: c745 b002 0000 0048 8d05 fa63 2a00 4889  .E.....H...c*.H.
+038f36d0: 45b8 48c7 45c0 0200 0000 488d 8548 ffff  E.H.E.....H..H..
+038f36e0: ff48 8945 c848 c745 d002 0000 0048 8d35  .H.E.H.E.....H.5
+038f36f0: 14a7 b601 488d 7da8 e883 e5ff ff0f 0b48  ....H.}........H
+038f3700: 8d3d 7adc bf01 ff17 48c7 4008 0000 0000  .=z.....H.@.....
+038f3710: e98f fdff ff48 8d3d ecbb b601 e82f 8f00  .....H.=...../..
+038f3720: 000f 0b66 6666 662e 0f1f 8400 0000 0000  ...ffff.........
 038f3730: 5548 89e5 803f 0074 0e48 8d05 f23f 2a00  UH...?.t.H...?*.
 038f3740: ba2b 0000 00eb 0c48 8d05 c83f 2a00 ba1c  .+.....H...?*...
 038f3750: 0000 0048 89f7 4889 c65d e951 b700 0090  ...H..H..].Q....
@@ -4578681,7 +4578681,7 @@
 045dd780: 2400 0000 fcc0 0200 a85b 31ff ffff ffff  $........[1.....
 045dd790: 4800 0000 0000 0000 0041 0e10 8602 430d  H........A....C.
 045dd7a0: 0643 8304 8e03 0000 2400 0000 24c1 0200  .C......$...$...
-045dd7b0: d05b 31ff ffff ffff a503 0000 0000 0000  .[1.............
+045dd7b0: d05b 31ff ffff ffff a303 0000 0000 0000  .[1.............
 045dd7c0: 0041 0e10 8602 430d 064a 8304 8e03 0000  .A....C..J......
 045dd7d0: 2400 0000 4cc1 0200 585f 31ff ffff ffff  $...L...X_1.....
 045dd7e0: 2f00 0000 0000 0000 0041 0e10 8602 430d  /........A....C.
    • boklm
      boklm @boklm ·

      Maybe comparing the obj-macos directory from two builds can help debug the issue.

      I have started a build with this patch: https://gitweb.torproject.org/user/boklm/tor-browser-build.git/commit/?h=bug_26475&id=1ceef284b016441783d276ba32542e9fcd9bfd75

      with the command:

      $ ./rbm/rbm build firefox --target alpha --target torbrowser-osx-x86_64

      And will upload the result when it finish.

    • boklm
      boklm @boklm ·

      Replying to boklm:

      Maybe comparing the obj-macos directory from two builds can help debug the issue.

      I have started a build with this patch: https://gitweb.torproject.org/user/boklm/tor-browser-build.git/commit/?h=bug_26475&id=1ceef284b016441783d276ba32542e9fcd9bfd75

      I pushed a fixup commit for that: https://gitweb.torproject.org/user/boklm/tor-browser-build.git/commit/?h=bug_26475&id=a5bb5818b3563a1a7a7d13a8068ef0e55943c13d

    • Georg Koppen
      Georg Koppen @gk ·
      Author

      I compared two obj-macos dirs and gkrust seems to be the culprit (as assumed).

      diff -u <(xxd tpo/gkrust-d3a9de07b53ab691.gkrust0.rcgu.o) <(xxd uninett/gkrust-d3a9de07b53ab691.gkrust0.rcgu.o)

      gives me among others

       00440e20: ff48 8b07 488b 4f08 4889 8d70 ffff ff48  .H..H.O.H..p...H
 00440e30: 8985 68ff ffff 488d 0500 0000 0048 8985  ..h...H......H..
 00440e40: 38ff ffff 48c7 8540 ffff ff06 0000 0048  8...H..@.......H
-00440e50: 8b3d 0000 0000 ff17 8078 2100 0f85 da01  .=.......x!.....
+00440e50: 8b3d 0000 0000 ff17 8078 2100 0f85 d801  .=.......x!.....
 00440e60: 0000 488b 3d00 0000 00ff 1780 7820 0075  ..H.=.......x .u
 00440e70: 1f48 8b3d 0000 0000 ff17 4889 c348 8d3d  .H.=......H..H.=
 00440e80: 0000 0000 4889 dee8 0000 0000 c643 2001  ....H........C .
-00440e90: 488b 3d00 0000 00ff 1748 8338 0174 6748  H.=......H.8.tgH
+00440e90: 488b 3d00 0000 00ff 1748 8338 0174 6548  H.=......H.8.teH
 00440ea0: 8b3d 0000 0000 ff17 660f 6f40 100f 57c9  .=......f.o@..W.
-00440eb0: 0f29 4810 b901 0000 0066 480f 6ec9 488b  .)H......fH.n.H.
-00440ec0: 0866 0f7f 0848 85c9 7429 6648 0f7e c348  .f...H..t)fH.~.H
-00440ed0: 85db 741f 660f 70c0 4e66 490f 7ec6 4889  ..t.f.p.NfI.~.H.
-00440ee0: df41 ff16 4983 7e08 0074 0848 89df e800  .A..I.~..t.H....
-00440ef0: 0000 0048 8b3d 0000 0000 ff17 4883 3801  ...H.=......H.8.
-00440f00: 0f85 8102 0000 488b 3d00 0000 00ff 1748  ......H.=......H
-00440f10: 8378 0800 7477 e800 0000 0048 89c3 4889  .x..tw.....H..H.
-00440f20: 9d48 ffff ff48 8b45 9048 8945 d048 8b45  .H...H.E.H.E.H.E
-00440f30: 8848 8945 c848 8b45 8048 8945 c048 8b85  .H.E.H.E.H.E.H..
-00440f40: 78ff ffff 4889 45b8 488b 8568 ffff ff48  x...H.E.H..h...H
-00440f50: 8b8d 70ff ffff 4889 4db0 4889 45a8 488d  ..p...H.M.H.E.H.
-00440f60: 7dd8 488d b548 ffff ff48 8d55 a8e8 0000  }.H..H...H.U....
-00440f70: 0000 f048 ff0b 0f85 a100 0000 488d bd48  ...H........H..H
-00440f80: ffff ffe8 0000 0000 e990 0000 0048 8b3d  .............H.=
-00440f90: 0000 0000 ff17 48c7 4008 ffff ffff 488d  ......H.@.....H.
-00440fa0: 4808 488d 5010 4889 9548 ffff ff48 898d  H.H.P.H..H...H..
-00440fb0: 50ff ffff 488b 7010 4885 f60f 84b0 0100  P...H.p.H.......
-00440fc0: 0048 8b3d 0000 0000 ff17 4889 c348 8b45  .H.=......H..H.E
-00440fd0: 9048 8945 d048 8b43 1848 8b4d 8848 894d  .H.E.H.C.H.M.H.M
-00440fe0: c848 8b4d 8048 894d c048 8b8d 78ff ffff  .H.M.H.M.H..x...
-00440ff0: 4889 4db8 488b 8d68 ffff ff48 8b95 70ff  H.M.H..h...H..p.
-00441000: ffff 4889 55b0 4889 4da8 488d 7dd8 488d  ..H.U.H.M.H.}.H.
-00441010: 55a8 ff50 3048 c743 0800 0000 008b 45d9  U..P0H.C......E.
-00441020: 8945 e88a 45d8 0fb7 4ddd 6689 4dec 8a4d  .E..E...M.f.M..M
-00441030: df88 4dee 3c04 0f85 8000 0000 e800 0000  ..M.<...........
-00441040: 0048 89c3 4889 9d48 ffff ff48 8b45 9048  .H..H..H...H.E.H
-00441050: 8945 d048 8b45 8848 8945 c848 8b45 8048  .E.H.E.H.E.H.E.H
-00441060: 8945 c048 8b85 78ff ffff 4889 45b8 488b  .E.H..x...H.E.H.
-00441070: 8568 ffff ff48 8b8d 70ff ffff 4889 4db0  .h...H..p...H.M.
-00441080: 4889 45a8 488d 7d98 488d b548 ffff ff48  H.E.H.}.H..H...H
-00441090: 8d55 a8e8 0000 0000 f048 ff0b 750c 488d  .U.......H..u.H.
-004410a0: bd48 ffff ffe8 0000 0000 807d 9803 7531  .H.........}..u1
-004410b0: 4881 c4c0 0000 005b 415e 5dc3 8845 9848  H......[A^]..E.H
-004410c0: 8b45 e08b 4de8 894d 990f b74d ec66 894d  .E..M..M...M.f.M
-004410d0: 9d8a 4dee 884d 9f48 8945 a080 7d98 0374  ..M..M.H.E..}..t
-004410e0: cf48 8b45 9848 8b4d a048 894d e048 8945  .H.E.H.M.H.M.H.E
-004410f0: d848 8d85 38ff ffff 4889 8548 ffff ff48  .H..8...H..H...H
-00441100: 8d05 0000 0000 4889 8550 ffff ff48 8d45  ......H..P...H.E
-00441110: d848 8985 58ff ffff 488d 0500 0000 0048  .H..X...H......H
-00441120: 8985 60ff ffff 488d 0500 0000 0048 8945  ..`...H......H.E
-00441130: a848 c745 b002 0000 0048 8d05 0000 0000  .H.E.....H......
-00441140: 4889 45b8 48c7 45c0 0200 0000 488d 8548  H.E.H.E.....H..H
-00441150: ffff ff48 8945 c848 c745 d002 0000 0048  ...H.E.H.E.....H
-00441160: 8d35 0000 0000 488d 7da8 e800 0000 000f  .5....H.}.......
-00441170: 0b48 8b3d 0000 0000 ff17 48c7 4008 0000  .H.=......H.@...
-00441180: 0000 e98f fdff ff48 8d3d 0000 0000 e800  .......H.=......
-00441190: 0000 000f 0b66 662e 0f1f 8400 0000 0000  .....ff.........
+00440eb0: 0f29 4810 b901 0000 0066 480f 6ec9 4883  .)H......fH.n.H.
+00440ec0: 3800 660f 7f08 7429 6648 0f7e c348 85db  8.f...t)fH.~.H..
+00440ed0: 741f 660f 70c0 4e66 490f 7ec6 4889 df41  t.f.p.NfI.~.H..A
+00440ee0: ff16 4983 7e08 0074 0848 89df e800 0000  ..I.~..t.H......
+00440ef0: 0048 8b3d 0000 0000 ff17 4883 3801 0f85  .H.=......H.8...
+00440f00: 8102 0000 488b 3d00 0000 00ff 1748 8378  ....H.=......H.x
+00440f10: 0800 7477 e800 0000 0048 89c3 4889 9d48  ..tw.....H..H..H
+00440f20: ffff ff48 8b45 9048 8945 d048 8b45 8848  ...H.E.H.E.H.E.H
+00440f30: 8945 c848 8b45 8048 8945 c048 8b85 78ff  .E.H.E.H.E.H..x.
+00440f40: ffff 4889 45b8 488b 8568 ffff ff48 8b8d  ..H.E.H..h...H..
+00440f50: 70ff ffff 4889 4db0 4889 45a8 488d 7dd8  p...H.M.H.E.H.}.
+00440f60: 488d b548 ffff ff48 8d55 a8e8 0000 0000  H..H...H.U......
+00440f70: f048 ff0b 0f85 a100 0000 488d bd48 ffff  .H........H..H..
+00440f80: ffe8 0000 0000 e990 0000 0048 8b3d 0000  ...........H.=..
+00440f90: 0000 ff17 48c7 4008 ffff ffff 488d 4808  ....H.@.....H.H.
+00440fa0: 488d 5010 4889 9548 ffff ff48 898d 50ff  H.P.H..H...H..P.
+00440fb0: ffff 488b 7010 4885 f60f 84b0 0100 0048  ..H.p.H........H
+00440fc0: 8b3d 0000 0000 ff17 4889 c348 8b45 9048  .=......H..H.E.H
+00440fd0: 8945 d048 8b43 1848 8b4d 8848 894d c848  .E.H.C.H.M.H.M.H
+00440fe0: 8b4d 8048 894d c048 8b8d 78ff ffff 4889  .M.H.M.H..x...H.
+00440ff0: 4db8 488b 8d68 ffff ff48 8b95 70ff ffff  M.H..h...H..p...
+00441000: 4889 55b0 4889 4da8 488d 7dd8 488d 55a8  H.U.H.M.H.}.H.U.
+00441010: ff50 3048 c743 0800 0000 008b 45d9 8945  .P0H.C......E..E
+00441020: e88a 45d8 0fb7 4ddd 6689 4dec 8a4d df88  ..E...M.f.M..M..
+00441030: 4dee 3c04 0f85 8000 0000 e800 0000 0048  M.<............H
+00441040: 89c3 4889 9d48 ffff ff48 8b45 9048 8945  ..H..H...H.E.H.E
+00441050: d048 8b45 8848 8945 c848 8b45 8048 8945  .H.E.H.E.H.E.H.E
+00441060: c048 8b85 78ff ffff 4889 45b8 488b 8568  .H..x...H.E.H..h
+00441070: ffff ff48 8b8d 70ff ffff 4889 4db0 4889  ...H..p...H.M.H.
+00441080: 45a8 488d 7d98 488d b548 ffff ff48 8d55  E.H.}.H..H...H.U
+00441090: a8e8 0000 0000 f048 ff0b 750c 488d bd48  .......H..u.H..H
+004410a0: ffff ffe8 0000 0000 807d 9803 7531 4881  .........}..u1H.
+004410b0: c4c0 0000 005b 415e 5dc3 8845 9848 8b45  .....[A^]..E.H.E
+004410c0: e08b 4de8 894d 990f b74d ec66 894d 9d8a  ..M..M...M.f.M..
+004410d0: 4dee 884d 9f48 8945 a080 7d98 0374 cf48  M..M.H.E..}..t.H
+004410e0: 8b45 9848 8b4d a048 894d e048 8945 d848  .E.H.M.H.M.H.E.H
+004410f0: 8d85 38ff ffff 4889 8548 ffff ff48 8d05  ..8...H..H...H..
+00441100: 0000 0000 4889 8550 ffff ff48 8d45 d848  ....H..P...H.E.H
+00441110: 8985 58ff ffff 488d 0500 0000 0048 8985  ..X...H......H..
+00441120: 60ff ffff 488d 0500 0000 0048 8945 a848  `...H......H.E.H
+00441130: c745 b002 0000 0048 8d05 0000 0000 4889  .E.....H......H.
+00441140: 45b8 48c7 45c0 0200 0000 488d 8548 ffff  E.H.E.....H..H..
+00441150: ff48 8945 c848 c745 d002 0000 0048 8d35  .H.E.H.E.....H.5
+00441160: 0000 0000 488d 7da8 e800 0000 000f 0b48  ....H.}........H
+00441170: 8b3d 0000 0000 ff17 48c7 4008 0000 0000  .=......H.@.....
+00441180: e98f fdff ff48 8d3d 0000 0000 e800 0000  .....H.=........
+00441190: 000f 0b66 6666 662e 0f1f 8400 0000 0000  ...ffff.........

      which is basically what we see in comment:1.

    • Georg Koppen
      Georg Koppen @gk ·
      Author

      Disabling stylo solves this for now. I'll leave this bug open for further investigation, though.

      Trac:
      Priority: Immediate to Very High
      Keywords: N/A deleted, GeorgKoppen201806 added

    • Georg Koppen
      Georg Koppen @gk ·
      Author

      Trac:
      Summary: ESR60-based .dmg images are not built reproducible to ESR60-based .dmg images are not built reproducibly with Stylo enabled

    • Georg Koppen
      Georg Koppen @gk ·
      Author

      As I expected this is a problem introduced by us switching to 1.26.1. Using version 1.25.0 of the Rust compiler XUL is still reproducible. So, to sum up:

      Starting with the switch to Rust 1.26.1 Tor Browser for macOS (and only for that platform) is not being built reproducibly anymore, once we enable Stylo. There are small differences visible in XUL stemming from gkrust (see: comment:1 and comment:4). Using either Rust 1.25.0 or disabling Stylo (what we are doing right now) is "solving" this problem.

      Alex, Manish: Do you know of any commit that could have caused this? If not, I can bisect I guess.

      Trac:
      Cc: boklm to boklm, manishearth@gmail.com, acrichton@mozilla.com
      Summary: ESR60-based .dmg images are not built reproducibly with Stylo enabled to ESR60-based .dmg images are not built reproducibly with Stylo enabled using rustc > 1.25.0
      Status: new to needs_information

    • Trac
      Trac @tracbot ·

      I know of https://github.com/rust-lang/rust/issues/47086 as an macOS-only deterministic compilation problem, but other than that I unfortunately don't know of any macOS-specific deterministic compilation issues as part of the toolchain itself.

      Trac:
      Username: alexcrichton

    • Georg Koppen
      Georg Koppen @gk ·
      Author

      Replying to alexcrichton:

      I know of https://github.com/rust-lang/rust/issues/47086 as an macOS-only deterministic compilation problem, but other than that I unfortunately don't know of any macOS-specific deterministic compilation issues as part of the toolchain itself.

      Okay, than let's try to track the issue down...

      Trac:
      Status: needs_information to assigned

    • Georg Koppen
      Georg Koppen @gk ·
      Author

      So, I tried for a while to get the bisecting going but it seems non-trivial. Alex: What is the recommended way of doing rust compiler bisecting? What I tried was using git bisect with the commit for 1.25.0 and 1.26.1, updating the sub modules and tarring the result up + using it on my build machines, but that breaks for different reasons (first one is no properly vendored crates are included (we compile with --enable-vendor); trying to get crates via crates.io-index fails with an SSL error...).

      Maybe easier: Are there somewhere nightly source tarballs available (like they are for the official releases) which I could test? I did not find any so far but that might already narrow down the problem sufficiently.

      Trac:
      Status: assigned to needs_information

    • Trac
      Trac @tracbot ·

      If you're ok downloading prebuilt artifacts I'd recommend bisection via nightlies which rustup can download. Otherwise there's nightly source tarballs as well at urls like https://static.rust-lang.org/dist/2018-06-29/rustc-nightly-src.tar.gz

      Trac:
      Username: alexcrichton

    • Trac
      Trac @tracbot ·

      There's also this tool, which is kinda like mozregression

      https://github.com/rust-lang-nursery/cargo-bisect-rustc

      It works off of the tarballs.

      I've never tried it with Firefox though.

      Trac:
      Username: manish.earth

    • Georg Koppen
      Georg Koppen @gk ·
      Author

      Okay, I tried the nightlies from 2018-02-25, 2018-03-28, and 2018-05-08 to find a regression range but to my surprise they all are good, meaning I get the same result on different build machines (which does not happen with stable versions > 1.25.0). So, it seems there is something in the stable code but not in the nightlies what is causing this which is confusing to me. We build with

      --enable-local-rust --enable-vendor --enable-extended --release-channel=stable --sysconfdir=etc --target=x86_64-apple-darwin --set=target.x86_64-apple-darwin.cc=x86_64-apple-darwin-clang

      but it seems that should not make a difference for nightly vs. stable builds. So, hrm...

    • Trac
      Trac @tracbot ·

      Oh dear that is indeed worrisome! The only difference between nightly and stable builds is --release-channel passed to ./configure so you should already be emulating our stable builds. It looks like you're cross-compiling rustc from Linux though? For us we compile natively on OSX and I wonder if that causes differences?

      Trac:
      Username: alexcrichton

    • Georg Koppen
      Georg Koppen @gk ·
      Author

      Replying to alexcrichton:

      Oh dear that is indeed worrisome! The only difference between nightly and stable builds is --release-channel passed to ./configure so you should already be emulating our stable builds. It looks like you're cross-compiling rustc from Linux though? For us we compile natively on OSX and I wonder if that causes differences?

      I avoided that by taking the respective nightly sources and compiled them as we compile the stable compiler.

      So, the plan right now is to take the nightly sources that are closest to 1.26.1 and remove parts of the diff until I find the problem.

    • Trac
      Trac @tracbot ·

      Ok sure yeah, you may want to go via git bisection in the repo perhaps? That'll need to download other sources but they're done via lock files and such so you shouldn't be downloading or using any different sources than we used to build the releases

      Trac:
      Username: alexcrichton

    • Georg Koppen
      Georg Koppen @gk ·
      Author

      Moving first batch of tickets to July 2018

      Trac:
      Keywords: TorBrowserTeam201806 deleted, TorBrowserTeam201807 added

    • Georg Koppen
      Georg Koppen @gk ·
      Author

      Moving my tickets.

      Trac:
      Keywords: GeorgKoppen201806 deleted, GeorgKoppen201807 added

    • Georg Koppen
      Georg Koppen @gk ·
      Author

      Okay, some good news here: I double-checked my setup and it seems I made a mistake which led to no bad nightly being found. I fixed that and the last nightly that is good is the one from March 07. Then there are no nightly sources between 03/08 and 03/14 (inclusive) and the nightly from March 15 is the first bad one.

      I fought a bit with using the repo for bisecting (I need to vendor all the crates before creating a tarball, making sure that I really have all the submodules initialized and updated, no .git dirs bundled but .gitmodules files available etc.) but it seems to work now. Bisecting...

    • Georg Koppen
      Georg Koppen @gk ·
      Author

      The first bad commit git bisect gives me is commit 2079a084df08c38eb4dbfc5c8de5c0245170c3d9 which is a merge. So, I need to dig deeper it seems...

    • Georg Koppen
      Georg Koppen @gk ·
      Author

      Bisecting seems tricky as I get intermittent differences which sometimes go away if one just tries several times. I've attached the disassmebly diff of gkrust-d3a9de07b53ab691.gkrust0.rcgu.o as it might help a bit narrowing things down. I wonder whether we actually got the differences due to an LLVM update...

    • Georg Koppen
      Georg Koppen @gk ·
      Author

      Trac:
      disassemble_gkrust-d3a9de07b53ab691.gkrust0.rcgu.o_diff

    • Trac
      Trac @tracbot ·

      Hm interesting! I wonder if this is perhaps related to https://github.com/rust-lang/rust/issues/52044? That claims it was fixed with the most recent LLVM upgrade. Are you able to reproduce the non-determinism on the most recent nightly?

      Trac:
      Username: alexcrichton

    • Georg Koppen
      Georg Koppen @gk ·
      Author

      Replying to alexcrichton:

      Hm interesting! I wonder if this is perhaps related to https://github.com/rust-lang/rust/issues/52044? That claims it was fixed with the most recent LLVM upgrade. Are you able to reproduce the non-determinism on the most recent nightly?

      Aha! That sounds promising and I certainly feel glandium's "This is driving me crazy", so this should be the issue then, right? ;)

      That said, I compiled the nightly from 2018-07-13 which should contain the LLVM upgrade and I can't reproduce the problem anymore. However, I can't either when compiling the one from from 2018-07-11 which should not contain the LLVM upgrade (it's based on commit e5f6498d3d5c9dac841009d7b49738923826af75). So, it seem the LLVM uprade (alone) is not enough to explain this bug, or am I missing something?

      Trying to figure out where all this started, I am pretty sure that 2018-02-15 is good and 2018-03-07 is bad.

    • Georg Koppen
      Georg Koppen @gk ·
      Author

      I wonder if building from scratch is important here (which I do and glandium did as well).

    • Georg Koppen
      Georg Koppen @gk ·
      Author

      Replying to gk:

      Replying to alexcrichton:

      Hm interesting! I wonder if this is perhaps related to https://github.com/rust-lang/rust/issues/52044? That claims it was fixed with the most recent LLVM upgrade. Are you able to reproduce the non-determinism on the most recent nightly?

      Aha! That sounds promising and I certainly feel glandium's "This is driving me crazy", so this should be the issue then, right? ;)

      That said, I compiled the nightly from 2018-07-13 which should contain the LLVM upgrade and I can't reproduce the problem anymore. However, I can't either when compiling the one from from 2018-07-11 which should not contain the LLVM upgrade (it's based on commit e5f6498d3d5c9dac841009d7b49738923826af75). So, it seem the LLVM uprade (alone) is not enough to explain this bug, or am I missing something?

      On the other hand testing your script from https://github.com/rust-lang/rust/issues/52044#issuecomment-402349038 shows easily that the one from 2018-07-13 is good while the one from 2018-07-11 is not. I guess I tried the Firefox build just not often enough with 2018-07-11...

      So, what's the suggested way to fix that? What until a new stable compiler with a fix for it is out (I guess that's 1.28.0) and use that one?

      Althought, tbh, it's scary to hope this is finally fixed without exactly knowing what the issue was. I might try to look closer at it.

    • Trac
      Trac @tracbot ·

      Ah yeah for the LLVM 7 change we're gonna let that ride the trains (aka not backport). If you can temporarily use a nightly that'd work but otherwise this may have to wait until that's released.

      Knowing for sure though what was causing the nondeterminism in LLVM would be great!

      Trac:
      Username: alexcrichton

    • Georg Koppen
      Georg Koppen @gk ·
      Author

      Replying to gk:

      Replying to alexcrichton:

      Hm interesting! I wonder if this is perhaps related to https://github.com/rust-lang/rust/issues/52044? That claims it was fixed with the most recent LLVM upgrade. Are you able to reproduce the non-determinism on the most recent nightly?

      [snip]

      Trying to figure out where all this started, I am pretty sure that 2018-02-15 is good and 2018-03-07 is bad.

      So, here is another interesting bit: While running the script (https://github.com/rust-lang/rust/issues/52044#issuecomment-402349038) on a Linux box does replicate the issue for a Linux build using the nightly from 2018-07-11, it does not replicate it using the nightly from 2018-03-07 (I ran the script a couple of times). On the other hand, using the nightly from 2018-03-07 to generate a Firefox build for macOS does show differences (while, as mentioned above, I seemingly can't reproduce the problem with the nightly from 2018-07-11 anymore when cross-compiling for macOS).

      To test better I adapted your repro script and added a respective --target x86_64-apple-darwin and with that setup it's easy to see the bug with the nightly from 2018-03-07.

      So, to sum up, the problem is not just bumping the LLVM version but it seems to be somewhat target dependent, too.

    • Georg Koppen
      Georg Koppen @gk ·
      Author

      Interesting, look what I get after trying to run the repro script for the macOS target after the bump to LLVM 7 (i.e. with the nightly from 2017-07-13):

      + for i in `seq 1 100`
+ rm -rf a b
+ mkdir a b
+ rustc /dev/stdin -O -C lto -C panic=abort -C codegen-units=1 --emit llvm-ir,obj --crate-type staticlib --out-dir a --target x86_64-apple-darwin
+ rustc /dev/stdin -O -C lto -C panic=abort -C codegen-units=1 --emit llvm-ir,obj --crate-type staticlib --out-dir b --target x86_64-apple-darwin
++ md5sum a/stdin.ll
++ awk '{print $1}'
+ a=d3665a14a5cf9bf1c4630c001e8d4dfc
++ md5sum b/stdin.ll
++ awk '{print $1}'
+ b=4d3336f9f9c5bc4b91653c1fc51d0bf6
+ '[' d3665a14a5cf9bf1c4630c001e8d4dfc '!=' 4d3336f9f9c5bc4b91653c1fc51d0bf6 ']'
+ echo IR is different
IR is different
+ exit 1

      Linux is fine (which is probably what glandium found).

      FWIW: I see different objects for macOS with the repro script when using the nightly from 2018-07-11, so, indeed, I did not do enough Firefox compilations to trigger the problem for macOS there.

    • Trac
      Trac @tracbot ·

      Hm fascinating! That's a good data point towards "maybe fixed" in LLVM but only accidentally for one platform rather than across the board?

      Trac:
      Username: alexcrichton

    • Georg Koppen
      Georg Koppen @gk ·
      Author

      Replying to alexcrichton:

      Hm fascinating! That's a good data point towards "maybe fixed" in LLVM but only accidentally for one platform rather than across the board?

      I actually think the issue glandium had and we have are not the same but probably related: the macOS one is happening with the upgrade to LLVM 6 (I am about to start bisecting those changes) while Linux is unaffected by that. But I'll have clues for the latter as well, but macOS first.

    • Georg Koppen
      Georg Koppen @gk ·
      Author

      Moving my tickets.

      Trac:
      Keywords: GeorgKoppen201807 deleted, GeorgKoppen201808 added

    • Georg Koppen
      Georg Koppen @gk ·
      Author

      Move our tickets to August.

      Trac:
      Keywords: TorBrowserTeam201807 deleted, TorBrowserTeam201808 added

    • Georg Koppen
      Georg Koppen @gk ·
      Author

      Alright, so, the problematic commit is 6b7b6b63a928479a29d9fc1282e553e409c66934. I tried to bisect LLVM to find out the bad revision. I compiled just LLVM separately and used --llvm-root to point to it during the rust build. It turns out that doing that, even with just the llvm code copied out of rust/src/llvm being on commit 6b7b6b63a928479a29d9fc1282e553e409c66934 is fine. However, double-checking, building LLVM during the usual rust build (i.e. without --llvm-root) still reproduces the bug.

      Thus, I can only think of two reasons causing the reprouducibility issue:

      1. The LLVM part is compiled differently within rust than I did. The compiler used is the same, though, not sure what other flags could cause this. I am doing
      cmake .. -G "Unix Makefiles" -DCMAKE_INSTALL_PREFIX=$distdir -DCMAKE_BUILD_TYPE:STRING=Release -DLLVM_INSTALL_UTILS=on $LLVM_HOME

      in the --llvm-root build which seems to be not overly fancy.

      1. There are additional pieces getting compiled in/used during the LLVM compilation done during the rust build that are causing the problem.

      I need to think more about how to proceed but, Alex, if you have any ideas that would be appreciated. :)

    • Trac
      Trac @tracbot ·

      Nice! Bisecting to the LLVM upgrade definitely makes sense to me. That was a massive LLVM upgrade though (from LLVM 4.0 to LLVM 6.0), so that would be quite the bisection range for a regression to be introduced in :(

      If it works, though, when you build LLVM yourself that's quite curious. The command we use to build LLVM though is pretty huge. Looking at one of our recent builds (https://travis-ci.org/rust-lang/rust/builds/411219658) the command we use on OSX is:

      "cmake" "/Users/travis/build/rust-lang/rust/src/llvm" "-DLLVM_ENABLE_ASSERTIONS=OFF" "-DLLVM_TARGETS_TO_BUILD=X86;ARM;AArch64;Mips;PowerPC;SystemZ;MSP430;Sparc;NVPTX;Hexagon" "-DLLVM_EXPERIMENTAL_TARGETS_TO_BUILD=WebAssembly;RISCV" "-DLLVM_INCLUDE_EXAMPLES=OFF" "-DLLVM_INCLUDE_TESTS=OFF" "-DLLVM_INCLUDE_DOCS=OFF" "-DLLVM_ENABLE_ZLIB=OFF" "-DWITH_POLLY=OFF" "-DLLVM_ENABLE_TERMINFO=OFF" "-DLLVM_ENABLE_LIBEDIT=OFF" "-DLLVM_ENABLE_LIBXML2=OFF" "-DLLVM_PARALLEL_COMPILE_JOBS=4" "-DLLVM_TARGET_ARCH=x86_64" "-DLLVM_DEFAULT_TARGET_TRIPLE=x86_64-apple-darwin" "-DLLVM_OCAML_INSTALL_PATH=usr/lib/ocaml" "-DCMAKE_EXE_LINKER_FLAGS=-static-libstdc++" "-DCMAKE_C_COMPILER=sccache" "-DCMAKE_C_COMPILER_ARG1=/Users/travis/build/rust-lang/rust/clang+llvm-6.0.0-x86_64-apple-darwin/bin/clang" "-DCMAKE_CXX_COMPILER=sccache" "-DCMAKE_CXX_COMPILER_ARG1=/Users/travis/build/rust-lang/rust/clang+llvm-6.0.0-x86_64-apple-darwin/bin/clang++" "-DCMAKE_C_FLAGS=-ffunction-sections -fdata-sections -fPIC --target=x86_64-apple-darwin -stdlib=libc++" "-DCMAKE_CXX_FLAGS=-ffunction-sections -fdata-sections -fPIC --target=x86_64-apple-darwin -stdlib=libc++" "-DCMAKE_INSTALL_PREFIX=/Users/travis/build/rust-lang/rust/build/x86_64-apple-darwin/llvm" "-DCMAKE_BUILD_TYPE=Release"

      I wonder if perhaps the way we compile LLVM is affecting this? Maybe some flag or maybe our own compiler we use on automation is introducing bugs? Or maybe it has to do with the C++ standard library and which is used?

      Trac:
      Username: alexcrichton

    • Georg Koppen
      Georg Koppen @gk ·
      Author

      Replying to alexcrichton:

      Nice! Bisecting to the LLVM upgrade definitely makes sense to me. That was a massive LLVM upgrade though (from LLVM 4.0 to LLVM 6.0), so that would be quite the bisection range for a regression to be introduced in :(

      If it works, though, when you build LLVM yourself that's quite curious. The command we use to build LLVM though is pretty huge. Looking at one of our recent builds (https://travis-ci.org/rust-lang/rust/builds/411219658) the command we use on OSX is:

      "cmake" "/Users/travis/build/rust-lang/rust/src/llvm" "-DLLVM_ENABLE_ASSERTIONS=OFF" "-DLLVM_TARGETS_TO_BUILD=X86;ARM;AArch64;Mips;PowerPC;SystemZ;MSP430;Sparc;NVPTX;Hexagon" "-DLLVM_EXPERIMENTAL_TARGETS_TO_BUILD=WebAssembly;RISCV" "-DLLVM_INCLUDE_EXAMPLES=OFF" "-DLLVM_INCLUDE_TESTS=OFF" "-DLLVM_INCLUDE_DOCS=OFF" "-DLLVM_ENABLE_ZLIB=OFF" "-DWITH_POLLY=OFF" "-DLLVM_ENABLE_TERMINFO=OFF" "-DLLVM_ENABLE_LIBEDIT=OFF" "-DLLVM_ENABLE_LIBXML2=OFF" "-DLLVM_PARALLEL_COMPILE_JOBS=4" "-DLLVM_TARGET_ARCH=x86_64" "-DLLVM_DEFAULT_TARGET_TRIPLE=x86_64-apple-darwin" "-DLLVM_OCAML_INSTALL_PATH=usr/lib/ocaml" "-DCMAKE_EXE_LINKER_FLAGS=-static-libstdc++" "-DCMAKE_C_COMPILER=sccache" "-DCMAKE_C_COMPILER_ARG1=/Users/travis/build/rust-lang/rust/clang+llvm-6.0.0-x86_64-apple-darwin/bin/clang" "-DCMAKE_CXX_COMPILER=sccache" "-DCMAKE_CXX_COMPILER_ARG1=/Users/travis/build/rust-lang/rust/clang+llvm-6.0.0-x86_64-apple-darwin/bin/clang++" "-DCMAKE_C_FLAGS=-ffunction-sections -fdata-sections -fPIC --target=x86_64-apple-darwin -stdlib=libc++" "-DCMAKE_CXX_FLAGS=-ffunction-sections -fdata-sections -fPIC --target=x86_64-apple-darwin -stdlib=libc++" "-DCMAKE_INSTALL_PREFIX=/Users/travis/build/rust-lang/rust/build/x86_64-apple-darwin/llvm" "-DCMAKE_BUILD_TYPE=Release"
}}}

I wonder if perhaps the way we compile LLVM is affecting this? Maybe some flag or maybe our own compiler we use on automation is introducing bugs? Or maybe it has to do with the C++ standard library and which is used?

      Well, I dont't know whether the problem we have is happening when using the "official" binaries. This is happening when cross-compiling rust ourselves for macOS. That said, I think I can exclude 1) from comment:33. I compiled LLVM with exactly the same arguments as used during the rust build (using the same runc container, compiler etc.) which are: {{{ cmake .. -G "Unix Makefiles" -DLLVM_ENABLE_ASSERTIONS=OFF -DLLVM_TARGETS_TO_BUILD="X86;ARM;AArch64;Mips;PowerPC;SystemZ;MSP430;Sparc;NVPTX;Hexagon" -DLLVM_EXPERIMENTAL_TARGETS_TO_BUILD=WebAssembly -DLLVM_INCLUDE_EXAMPLES=OFF -DLLVM_INCLUDE_TESTS=OFF -DLLVM_INCLUDE_DOCS=OFF -DLLVM_ENABLE_ZLIB=OFF -DWITH_POLLY=OFF -DLLVM_ENABLE_TERMINFO=OFF -DLLVM_ENABLE_LIBEDIT=OFF -DLLVM_PARALLEL_COMPILE_JOBS=4 -DLLVM_TARGET_ARCH=x86_64 -DLLVM_DEFAULT_TARGET_TRIPLE=x86_64-unknown-linux-gnu -DLLVM_LINK_LLVM_DYLIB=ON -DCMAKE_C_COMPILER=cc -DCMAKE_CXX_COMPILER=c++ -DCMAKE_C_FLAGS="-ffunction-sections -fdata-sections -fPIC -m64" -DCMAKE_CXX_FLAGS="-ffunction-sections -fdata-sections -fPIC -m64" -DCMAKE_INSTALL_PREFIX=$distdir -DCMAKE_BUILD_TYPE:String=Release -DLLVM_INSTALL_UTILS=on $LLVM_HOME

      (`-DLLVM_INSTALL_UTILS=on` is only used when compiling LLVM outside of the rust compilation but I doubt this would make a difference with respect to this bug)

and used it with `--llvm-root` and the test script is running fine.

Which leaves 2). I reverted the `dlmalloc` and `libcompiler_builtins/compiler-rt` submodule updates but compiling LLVM during the rust build still exhibits the failing test script. Thus, I think those updates does not cause this bug. I wonder what else is different between the in-tree LLVM compilation and the one outside of it (I am just tarring up src/llvm, excluding `.git`, using that tarball for compilation)...
    • Trac
      Trac @tracbot ·

      Hm so if LLVM is compiled exactly the same way that is indeed a little worrisome. One other thing we do is that if we're compiling an in-tree LLVM we specify the LLVM_RUSTLLVM define for C++ code (here - https://github.com/rust-lang/rust/blob/bf1e461173e3936e4014cc951dfbdd7d9ec9190b/src/bootstrap/compile.rs#L736-L740). That's used for a few minor features but none that should touch the reproducibility of the build in theory...

      I'm running out of ideas unfortunately :(

      Trac:
      Username: alexcrichton

    • Georg Koppen
      Georg Koppen @gk ·
      Author

      Not setting LLVM_RUSTLLVM does not change things as you expected. So, I could do the LLVM bisecting with LLVM built during the rust compilation to give us a better hint at what is going wrong. However, I'd like to avoid that as it's probably a tedious pain in the rear and while the LLVM upgrade is triggering the bug it's probably nothing we should fix on the LLVM side. I'll think about something smarter and will go back to this bug after we got Tor Browser 8 out.

      We might stick to just building LLVM outside of rust for macOS cross-compilation for the time being to move around this issue.

      Trac:
      Status: needs_information to new

    • Georg Koppen
      Georg Koppen @gk ·
      Author

      Moving my tickets to September.

      Trac:
      Keywords: GeorgKoppen201808 deleted, GeorgKoppen201809 added

    • Georg Koppen
      Georg Koppen @gk ·
      Author

      Moving our tickets to September 2018

      Trac:
      Keywords: TorBrowserTeam201808 deleted, TorBrowserTeam201809 added

    • Georg Koppen
      Georg Koppen @gk ·
      Author

      Moving tickets to October

      Trac:
      Keywords: TorBrowserTeam201809 deleted, TorBrowserTeam201810 added

    • Georg Koppen
      Georg Koppen @gk ·
      Author

      #27937 (moved) is a duplicate: We have similar issues on Linux even though it seems they are even harder to reproduce. I assume for now the underlying issue is the same for both platforms.

      Trac:
      Description: Contrary to our testing results the .dmg bundles we produce for 8.0a9 are differing on two machines.

      to

      Contrary to our testing results the .dmg bundles we produce for 8.0a9 are differing on two machines. And it turns out Linux bundles are affected as well (see: #27937 (moved) for the original Linux bug report).
      Summary: ESR60-based .dmg images are not built reproducibly with Stylo enabled using rustc > 1.25.0 to ESR60-based Tor Browser bundles are not built reproducibly with Stylo enabled using rustc > 1.25.0
      Priority: Very High to Immediate

    • Georg Koppen
      Georg Koppen @gk ·
      Author

      Moving my tickets to October.

      Trac:
      Keywords: GeorgKoppen201809 deleted, GeorgKoppen201810 added

    • Georg Koppen
      Georg Koppen @gk ·
      Author

      Okay, a further piece in the puzzle: The problem goes away if I drop -C lto in the test script and only use

      rustc /dev/stdin -O -C panic=abort -C codegen-units=1 --emit llvm-ir,obj --crate-type staticlib --out-dir a --target x86_64-apple-darwin

      I tried that because I saw

                  if self.config.rust_codegen_units.is_none() &&
               self.build.is_rust_llvm(compiler.host) &&
               self.config.rust_thinlto {
                cargo.env("RUSTC_THINLTO", "1");
            } else if self.config.rust_codegen_units.is_none() {
                // Generally, if ThinLTO has been disabled for some reason, we
                // want to set the codegen units to 1. However, we shouldn't do
                // this if the option was specifically set by the user.
                cargo.env("RUSTC_CODEGEN_UNITS", "1");
            }

      in builder.rs.

      I wonder how bad it would be if we just dropped that when building Rust code (see: https://dxr.mozilla.org/mozilla-esr60/source/config/rules.mk#863) while trying to track the underlying issue down.

    • Georg Koppen
      Georg Koppen @gk ·
      Author

      Replying to gk:

      1. There are additional pieces getting compiled in/used during the LLVM compilation done during the rust build that are causing the problem.

      That's what's happening in this case. The relevant part is

                  if self.config.rust_codegen_units.is_none() &&
               self.build.is_rust_llvm(compiler.host) &&
               self.config.rust_thinlto {
                cargo.env("RUSTC_THINLTO", "1");
            } else if self.config.rust_codegen_units.is_none() {
                // Generally, if ThinLTO has been disabled for some reason, we
                // want to set the codegen units to 1. However, we shouldn't do
                // this if the option was specifically set by the user.
                cargo.env("RUSTC_CODEGEN_UNITS", "1");
            }

      in builder.rs.

      For some reason RUSTC_THINLTO is only set if one does not specify a LLVM with --llvm-root and that is the difference I hit. With it set the reproducibility problem emerges (even for LLVMs provided by --llvm-root if I rip out self.build.is_rust_llvm(compiler.host) &&) without it not.

      So, to sum up so far: Compiling with -C lto is fine unless RUSTC_THINLTO is used for compiling rust for macOS. Or to be more precise: unless RUSTC_THINLTO is used for the apple target libstd and related libraries. I could not pinpoint the exact lib that is causing this issue, though, yet (I can't easily replace them one by one as otherwise rustc is complaining about libstd being in need of recompilation).

      I guess more bisecting is next. :) Alex, do you think we could just avoid setting RUSTC_THINLTO for now when compiling the rust compiler? Or does that have any serious, known downsides?

    • Trac
      Trac @tracbot ·

      Oh sorry about this, that's a good discovery! Those settings definitely engage a lot of LLVM infrastructure that's not otherwise engaged, which could help explain why something nondeterministic is coming out the other end.

      The settings in bootstrap are pretty confusing, but what's happening here is either rustc is compiled with 16 codegen units (each crate turns into 16 object files) which are then optimized as a set with ThinLTO or each compiler crate is compiled into one object file and no ThinLTO is used.

      To clarify is the compiler's own binary nondeterministic when the compiler's crates are built with 16 CGUs + ThinLTO? Or is the compiler's binary deterministic but its output deterministic?

      In terms of impact of these settings:

      • 1 CGU builds are basically equivalent to 16 CGUs + ThinLTO. The 1 CGU build is slower to compile the compiler (less opportunities for parallelism), but is likely 1%-ish faster than the 16 CGUs + ThinLTO.
      • The 16 CGUs + ThinLTO is the default setting (as you've found)
      • If 16 CGUs are used without ThinLTO, the resulting compiler is probably horrendously slow (lots of missed inlining opportunities).

      Or put another way, if you disable ThinLTO you'll want to be sure to also compile with one codegen unit, which should happen in rustbuild via the above block.

      Trac:
      Username: alexcrichton

    • Georg Koppen
      Georg Koppen @gk ·
      Author

      Replying to alexcrichton:

      Oh sorry about this, that's a good discovery! Those settings definitely engage a lot of LLVM infrastructure that's not otherwise engaged, which could help explain why something nondeterministic is coming out the other end.

      The settings in bootstrap are pretty confusing, but what's happening here is either rustc is compiled with 16 codegen units (each crate turns into 16 object files) which are then optimized as a set with ThinLTO or each compiler crate is compiled into one object file and no ThinLTO is used.

      To clarify is the compiler's own binary nondeterministic when the compiler's crates are built with 16 CGUs + ThinLTO? Or is the compiler's binary deterministic but its output deterministic?

      When the compiler's crates (in fact just the macOS libstd + code it depends on) are built with 16 CGUs + ThinLTO its output is non-deterministic but only if I use -C lto (if I drop -C lto all is fine, too). The binary might, too, be non-deterministic, I have not checked. We are currently not concerned with getting a reproducibly built rust compiler. Right now, just the output that compiler gives us matters.

      In terms of impact of these settings:

      • 1 CGU builds are basically equivalent to 16 CGUs + ThinLTO. The 1 CGU build is slower to compile the compiler (less opportunities for parallelism), but is likely 1%-ish faster than the 16 CGUs + ThinLTO.
      • The 16 CGUs + ThinLTO is the default setting (as you've found)
      • If 16 CGUs are used without ThinLTO, the resulting compiler is probably horrendously slow (lots of missed inlining opportunities).

      Or put another way, if you disable ThinLTO you'll want to be sure to also compile with one codegen unit, which should happen in rustbuild via the above block.

      Okay, thanks for those explanations, really helpful.

    • Trac
      Trac @tracbot ·

      One way I might be able to still help as well is narrowing down where the nondeterminism is introduced. If you use -C save-temps when compiling the compiler should spew dozens of files all over the place. Each of these files in theory represents the various stages of compilation and provides snapshots into the compiler's pipeline. If you could find the set of files that are nondeterministic (we know it's at least the object files!) then that may help narrow this down as well!

      If a 16 CGU libstd + full crate LTO is the issue it sounds like this may be an issue with the LLVM "linker", but that's just a guess!

      Trac:
      Username: alexcrichton

    • Georg Koppen
      Georg Koppen @gk ·
      Author

      I have a patch that sets codegen-units to 1 which seems the way to disable Thin LTO. Alas, there is no direct option for disabling LTO yet in 1.26.1. Moreover, using the --set configure option to adjust the codegen-units does not seem to work as the value gets interpreted as a string but we need an integer. So, I resorted to good ol' sed.

      I made some testbuilds and tested the compiler against the adapted script from https://github.com/rust-lang/rust/issues/52044#issuecomment-402349038 and I got matching builds and the script is happy now.

      bug_26475_v2 (https://gitweb.torproject.org/user/gk/tor-browser-build.git/commit/?h=bug_26475_v2&id=e2da10a29f46762a8561e30c36915f8568d18c71) has the proposed workaround.

      Trac:
      Status: new to needs_review
      Keywords: TorBrowserTeam201810 deleted, TorBrowserTeam201810R added

    • Georg Koppen
      Georg Koppen @gk ·
      Author

      Replying to gk:

      I made some testbuilds and tested the compiler against the adapted script from https://github.com/rust-lang/rust/issues/52044#issuecomment-402349038 and I got matching builds and the script is happy now.

      FWIW: I am actually not sure if I solved #27937 (moved) with the patch as I was never able to reproduce the Linux issue. Looking at the diff my best guess is that it might not be panic but still LTO related. And the idea is to have the patch up for review squashing both cases.

    • Trac
      Trac @tracbot ·

      Glad that ended up working out! From my testing oh-so-long-ago at this point I suspected this was an LLVM issue, and disabling ThinLTO effectively serializes all of LLVM's work, so I wonder if this is a race somewhere in LLVM between some shared data structure across threads which affects ordering...

      Trac:
      Username: alexcrichton

    • Georg Koppen
      Georg Koppen @gk ·
      Author

      Replying to alexcrichton:

      Glad that ended up working out! From my testing oh-so-long-ago at this point I suspected this was an LLVM issue, and disabling ThinLTO effectively serializes all of LLVM's work, so I wonder if this is a race somewhere in LLVM between some shared data structure across threads which affects ordering...

      Maybe. I am still bisecting LLVM (what introduced the issue and what fixed it as the latter is not clear either) which is... non-trivial. I'll update this ticket with my findings.

    • boklm
      boklm @boklm ·

      Replying to gk:

      bug_26475_v2 (https://gitweb.torproject.org/user/gk/tor-browser-build.git/commit/?h=bug_26475_v2&id=e2da10a29f46762a8561e30c36915f8568d18c71) has the proposed workaround.

      After doing two separates testbuild using this branch, I got two non matching TorBrowser-8.5a3-osx64_en-US.dmg files. After checking the content of the dmg files, only the snowflake files are different, so this is probably #27827 (moved). So it seems the issue from this ticket is fixed.

      However, I also got two non matching mar-tools-mac64.zip files. https://people.torproject.org/~boklm/tmp/bug_26475/1/mar-tools-mac64.zip.asc https://people.torproject.org/~boklm/tmp/bug_26475/1/mar-tools-mac64.zip https://people.torproject.org/~boklm/tmp/bug_26475/2/mar-tools-mac64.zip.asc https://people.torproject.org/~boklm/tmp/bug_26475/2/mar-tools-mac64.zip

      I don't know if the non-matching mar-tools is caused by this patch, or something else.

    • boklm
      boklm @boklm ·

      Replying to boklm:

      However, I also got two non matching mar-tools-mac64.zip files. https://people.torproject.org/~boklm/tmp/bug_26475/1/mar-tools-mac64.zip.asc https://people.torproject.org/~boklm/tmp/bug_26475/1/mar-tools-mac64.zip https://people.torproject.org/~boklm/tmp/bug_26475/2/mar-tools-mac64.zip.asc https://people.torproject.org/~boklm/tmp/bug_26475/2/mar-tools-mac64.zip

      If you want to compare with your build, those mar-tools-mac64.zip have been generated with make testbuild on branch bug_26475_v2, with testbuild configured to do alpha builds.

    • boklm
      boklm @boklm ·

      Other than this issue with mar-tools-mac64.zip (which seems unrelated), the patch looks good to me.

    • Georg Koppen
      Georg Koppen @gk ·
      Author

      Replying to boklm:

      Other than this issue with mar-tools-mac64.zip (which seems unrelated), the patch looks good to me.

      I opened #28102 (moved) for that.

      Thanks for the review and, Alex, thanks for your help! I merged the patch to master (commit 0166d3d70e81c6c1072e03f6b950c8b2eb181343), marking it for backport as we want Stylo enabled for macOS and potentially fix the reproducibility issue on Linux.

      I'll update this ticket with my bisecting results once I am done, so that we have a chance to understand what actually went wrong and how it got fixed later on.

      Trac:
      Status: needs_review to closed
      Keywords: N/A deleted, tbb-backport added
      Resolution: N/A to fixed

    • Georg Koppen
      Georg Koppen @gk ·
      Author

      Finally, after bisecting compiler crashes (aka https://bugs.llvm.org/show_bug.cgi?id=33917), too, I got the problematic commit. It is r304593 (together with r301649). Let me know if I can be of any help if that does not make sense on first glance. :) I am happy to help understanding what is going wrong here.

      That said, I'll start figuring out why the issue vanished as I could not reproduce it anymore taking a nightly from mid-September (I found that different IRs showed up after the switch to LLVM 7, see: comment:28 which is replacing the issue in this bug with another reproducibility problem).

    • Trac
      Trac @tracbot ·

      The first commit you mentioned - https://reviews.llvm.org/D33320 - looks like it's just some minor renamings? The second though - https://reviews.llvm.org/D32653 - definitely looks more suspicious. Since the second commit is turning a flag on by default, could you take a "good LLVM" just before that commit, turning the flag on, and seeing if it has the same reproducibility issue?

      Otherwise, do you have a standalone LLVM test case you were testing with? Or was it largely always through rustc? I don't mind helping out to file a bug in LLVM!

      Trac:
      Username: alexcrichton

    • Georg Koppen
      Georg Koppen @gk ·
      Author

      Replying to alexcrichton:

      The first commit you mentioned - https://reviews.llvm.org/D33320 - looks like it's just some minor renamings?

      Oops, right. That's the last good commit. The problem is r304594 (which is enabling r301649).

      The second though - https://reviews.llvm.org/D32653 - definitely looks more suspicious. Since the second commit is turning a flag on by default, could you take a "good LLVM" just before that commit, turning the flag on, and seeing if it has the same reproducibility issue?

      I can do that with r305493 (i.e. the last good commit) but given that r305494 is just flipping that feature on by default (and removing unused code) I doubt this will give us new insights. Let me know, though, if you think otherwise.

      Otherwise, do you have a standalone LLVM test case you were testing with? Or was it largely always through rustc? I don't mind helping out to file a bug in LLVM!

      I don't have a standalone LLVM testcase, alas. Not sure how I would write a proper one for this usecase. :( I bisected through rustc adapting your test script in https://github.com/rust-lang/rust/issues/52044#issuecomment-402349038.

    • Trac
      Trac @tracbot ·

      Ok so just to make sure I understand, LLVM is completely deterministic up to and including r304593 - https://reviews.llvm.org/D33320. When you go one more commit to r304594 - https://reviews.llvm.org/D32653 - this LLVM is no longer deterministic. The commit in question here that makes LLVM nondeterministic is enabling r301649 - https://reviews.llvm.org/D31085 - a heuristic for something.

      This was all tested with an adapted script, where you're compiling a fixed version of rustc against a varying version of LLVM. The rustc linked with LLVM from r304593 is deterministic and the rustc linked with LLVM form r304594 is nondeterministic. The script is then an adaptation of the comment you mentioned.

      Does that all sound right? If so I think that's definitely enough to open an issue on LLVM itself and try to get the ball rolling there! It'd be maximally useful to have a standalone test case, but I sort of suspect this may not have an easy standalone test case and is related to how rustc is using LLVM on multiple threads internally, which LLVM's CLI tools don't do.

      Trac:
      Username: alexcrichton

    • Trac
      Trac @tracbot ·

      Oh so if that's all true, another question, what version of rustc is this using?

      And additionally, y'all are seeing reproducibility issues on the current nightly release of Rust, right? (latest LLVM we're using as well as latest rustc). IIRC though those issues haven't been reduced yet and it's suspected that this one is the cause?

      Trac:
      Username: alexcrichton

    • Georg Koppen
      Georg Koppen @gk ·
      Author

      Replying to alexcrichton:

      Ok so just to make sure I understand, LLVM is completely deterministic up to and including r304593 - https://reviews.llvm.org/D33320. When you go one more commit to r304594 - https://reviews.llvm.org/D32653 - this LLVM is no longer deterministic. The commit in question here that makes LLVM nondeterministic is enabling r301649 - https://reviews.llvm.org/D31085 - a heuristic for something.

      Yes.

      This was all tested with an adapted script, where you're compiling a fixed version of rustc against a varying version of LLVM. The rustc linked with LLVM from r304593 is deterministic and the rustc linked with LLVM form r304594 is nondeterministic. The script is then an adaptation of the comment you mentioned.

      Yes.

    • Georg Koppen
      Georg Koppen @gk ·
      Author

      Replying to alexcrichton:

      Oh so if that's all true, another question, what version of rustc is this using?

      The one at 6b7b6b63a928479a29d9fc1282e553e409c66934 with a bunch of patches for the LLVM bisecting.

      And additionally, y'all are seeing reproducibility issues on the current nightly release of Rust, right? (latest LLVM we're using as well as latest rustc). IIRC though those issues haven't been reduced yet and it's suspected that this one is the cause?

      I think we could be lucky in that it got fixed more or less recently during the upgrade to LLVM 7. The result is not conclusive yet as there popped another reproducibility issue up (see: comment:28).

      So, I'd suggest waiting with filing an LLVM bug until I have bisected that other issue.

    • Trac
      Trac @tracbot ·

      Ok, sounds good to me!

      Trac:
      Username: alexcrichton

    • Georg Koppen
      Georg Koppen @gk ·
      Author

      Backported to our stable branch (maint-8.0) (commit dd1d00a2fed85c8bb22c4f85bbc45da00d0e8c05).

      Trac:
      Keywords: tbb-backport deleted, tbb-backported added

    • Trac closed

      closed

    • Georg Koppen mentioned in issue #27443 (moved)

      mentioned in issue #27443 (moved)

    • Georg Koppen mentioned in issue #27937 (moved)

      mentioned in issue #27937 (moved)

    • Georg Koppen mentioned in issue #28157 (moved)

      mentioned in issue #28157 (moved)

    • Georg Koppen mentioned in issue #28240 (moved)

      mentioned in issue #28240 (moved)

    • boklm mentioned in issue #28260 (moved)

      mentioned in issue #28260 (moved)

    • Georg Koppen mentioned in issue #31860 (moved)

      mentioned in issue #31860 (moved)

    • Georg Koppen mentioned in issue #32052 (moved)

      mentioned in issue #32052 (moved)

    • Georg Koppen mentioned in issue #32053 (moved)

      mentioned in issue #32053 (moved)

    • Trac moved to tpo/applications/tor-browser#26475 (closed)

      moved to tpo/applications/tor-browser#26475 (closed)

    Please register or sign in to reply