Android reproducible build of Snowflake
This ticket is for tracking the progress of porting Snowflake to Android.
Activity
added GeorgKoppen201904 in Legacy / Trac TorBrowserTeam202005R in Legacy / Trac component::applications/tor browser in Legacy / Trac ex-sponsor-19 in Legacy / Trac owner::cohosh in Legacy / Trac parent::30318 in Legacy / Trac priority::medium in Legacy / Trac resolution::fixed in Legacy / Trac reviewer::gk in Legacy / Trac severity::normal in Legacy / Trac sponsor::28-must in Legacy / Trac status::closed in Legacy / Trac tbb-10.0a1 in Legacy / Trac tbb-mobile in Legacy / Trac tbb-rbm in Legacy / Trac type::project in Legacy / Trac labels
Hans-Christoph did the bulk of porting both go-webrtc and Snowflake to Android. The process is narrated in legacy/trac#28205 (moved) and culminated in
- https://github.com/keroserene/go-webrtc/pull/90 (go-webrtc port to Android)
- https://github.com/keroserene/snowflake/pull/43 (Snowflake port to Android and deb packages)
The current status, as I understand it, is that the build is reproducible, but includes or at least downloads the proprietary Google Play Services library as a side effect of the build.
Trac:
Cc: dcf, arlolra to dcf, arlolra, eighthave
Trac:
Parent: N/A to legacy/trac#19001 (moved)Replying to dcf:
- https://github.com/keroserene/go-webrtc/pull/90 (go-webrtc port to Android)
The current status, as I understand it, is that the build is reproducible, but includes or at least downloads the proprietary Google Play Services library as a side effect of the build.
The android changes are now merged into go-webrtc. We found a workaround for the Google Play services problem.
You can see build instructions for various platforms by looking at the
.gitlab-ci.ymlfile: https://github.com/keroserene/snowflake/blob/master/.gitlab-ci.ymlThese will build the Go code using the libwebrtc binaries included in the git repo.
-
Oops, I guess my
.gitlab-ci.ymlnever got merged upstream, you can see the android build in my fork: https://gitlab.com/eighthave/snowflake/blob/master/.gitlab-ci.yml Replying to gk:
dcf: do you have a script/instructions somewhere on how to build the whole thing which we could then translate into our
tor-browser-buildprocess? If you plan to do that work yourself, you are more than welcome! :). We'll likely put that on our plate otherwise.I didn't do anything beyond making sure that go-webrtc and snowflake would compile for Android, because I don't know how the pluggable transport integration works on Android. eighthave was showing us a
gomobile bindsetup that doesn't use separate executables (requires patching outpackage main) and I didn't know if that was how it was going to be done.I think that go-webrtc will only require minor changes in projects/webrtc/config and projects/snowflake/config. Here you can see the changes we made in the upstream build script: https://github.com/keroserene/go-webrtc/pull/90/files#diff-0b83f9dedf40d7356e5ca147a077acb4 In projects/webrtc/config:
- Set
TARGET_OS=androidTARGET_CPU=arm - Append
target_os = [ 'android' ]to .gclient. I think it's safe to do across all targets, as it only affects what gets downloaded. - In
gclient sync,echo nto not agree to the proprietary Google Play services license (also prevents the Play libraries from downloading). You could alternately patch out this check, etc. - In
gn gen, setsymbol_level=0. This is needed to actually disable debugging symbols; otherwise the libwebrtc is like 100 MB. - Upstream we are using Google's prebuilt arm-linux-androideabi-ar, but it would be better to build your own. In projects/snowflake/config:
- Set
GOOS=androidGOARCH=arm. - You may have to copy webrtc-android-armeabi-v7a.pc over webrtc-linux-arm.pc, as in eighthave's script.
- Set
Replying to dcf:
Replying to gk:
dcf: do you have a script/instructions somewhere on how to build the whole thing which we could then translate into our
tor-browser-buildprocess? If you plan to do that work yourself, you are more than welcome! :). We'll likely put that on our plate otherwise.I didn't do anything beyond making sure that go-webrtc and snowflake would compile for Android, because I don't know how the pluggable transport integration works on Android. eighthave was showing us a
gomobile bindsetup that doesn't use separate executables (requires patching outpackage main) and I didn't know if that was how it was going to be done.I think that go-webrtc will only require minor changes in projects/webrtc/config and projects/snowflake/config. Here you can see the changes we made in the upstream build script: https://github.com/keroserene/go-webrtc/pull/90/files#diff-0b83f9dedf40d7356e5ca147a077acb4 In projects/webrtc/config:
- Set
TARGET_OS=androidTARGET_CPU=arm - Append
target_os = [ 'android' ]to .gclient. I think it's safe to do across all targets, as it only affects what gets downloaded. - In
gclient sync,echo nto not agree to the proprietary Google Play services license (also prevents the Play libraries from downloading). You could alternately patch out this check, etc. - In
gn gen, setsymbol_level=0. This is needed to actually disable debugging symbols; otherwise the libwebrtc is like 100 MB. - Upstream we are using Google's prebuilt arm-linux-androideabi-ar, but it would be better to build your own. In projects/snowflake/config:
- Set
GOOS=androidGOARCH=arm. - You may have to copy webrtc-android-armeabi-v7a.pc over webrtc-linux-arm.pc, as in eighthave's script.
Thanks, that's helpful.
- Set
-
Trac:
Parent: legacy/trac#19001 (moved) to legacy/trac#30318 (moved) 
There is a patch for review in branch
bug_28672_v6: https://gitweb.torproject.org/user/boklm/tor-browser-build.git/commit/?h=bug_28672_v6&id=c18a7c37c9a439205da55830cbecc0c4a89429ecI did not check yet that the build is reproducible.
Trac:
Status: new to needs_review
Keywords: TorBrowserTeam201906 deleted, TorBrowserTeam201906R added-
Replying to boklm:
There is a patch for review in branch
bug_28672_v6: https://gitweb.torproject.org/user/boklm/tor-browser-build.git/commit/?h=bug_28672_v6&id=c18a7c37c9a439205da55830cbecc0c4a89429ecI did not check yet that the build is reproducible.
They are not but it seems we are close. Just
libgojni.sodiffers for me with what seems to be mostly different tmp directories being used during build:--- /dev/fd/63 2019-06-14 18:21:34.285763344 +0200 +++ /dev/fd/62 2019-06-14 18:21:34.285763344 +0200 @@ -20,12 +20,12 @@ 00000130: 0400 0000 52e5 7464 f089 0d00 f099 0d00 ....R.td........ 00000140: f099 0d00 1066 0a00 1066 0a00 0600 0000 .....f...f...... 00000150: 0800 0000 0400 0000 5300 0000 0400 0000 ........S....... -00000160: 476f 0000 7043 6a4a 6c57 3058 4d41 5478 Go..pCjJlW0XMATx -00000170: 5970 6c6f 3775 3452 2f44 6844 5253 6a69 Yplo7u4R/DhDRSji -00000180: 6434 6435 525f 6b79 6d34 715a 462f 2d74 d4d5R_kym4qZF/-t -00000190: 4275 6253 4632 2d6a 344a 714c 3052 6448 BubSF2-j4JqL0RdH -000001a0: 6568 2f69 7a52 4435 5f67 3931 775f 4564 eh/izRD5_g91w_Ed -000001b0: 6d6b 497a 5075 3700 0000 0000 0000 0000 mkIzPu7......... +00000160: 476f 0000 786a 5532 4a43 634c 694f 3272 Go..xjU2JCcLiO2r +00000170: 6b33 5172 4356 586e 2f5f 4d2d 384a 7a59 k3QrCVXn/_M-8JzY +00000180: 3536 4178 354d 6d53 5f5a 784f 612f 5858 56Ax5MmS_ZxOa/XX +00000190: 7337 314b 6d44 5672 4f33 796e 3973 3532 s71KmDVrO3yn9s52 +000001a0: 5076 2f49 4f4c 5378 6b67 6d6e 395f 4665 Pv/IOLSxkgmn9_Fe +000001b0: 722d 7033 2d45 6f00 0000 0000 0000 0000 r-p3-Eo......... 000001c0: 0000 0000 0000 0000 0100 0000 0000 0000 ................ 000001d0: 0000 0000 1200 0000 4300 0000 0000 0000 ........C....... 000001e0: 0000 0000 1200 0000 5000 0000 48d9 0b00 ........P...H... @@ -133,7 +133,7 @@ 00000840: 0000 0000 1000 f1ff 005f 5f63 7861 5f66 .........__cxa_f 00000850: 696e 616c 697a 6500 4c49 4243 006c 6962 inalize.LIBC.lib 00000860: 632e 736f 002f 746d 702f 676f 2d62 7569 c.so./tmp/go-bui -00000870: 6c64 3235 3637 3734 3035 332f 6230 3031 ld256774053/b001 +00000870: 6c64 3739 3430 3230 3539 332f 6230 3031 ld794020593/b001 00000880: 2f65 7865 2f61 2e6f 7574 005f 5f63 7861 /exe/a.out.__cxa 00000890: 5f61 7465 7869 7400 5f63 676f 5f38 3938 _atexit._cgo_898 000008a0: 6432 6164 3234 6165 315f 4366 756e 635f d2ad24ae1_Cfunc_ @@ -330,7 +330,7 @@ 00001490: 0200 0200 0200 0200 0200 0200 0100 0100 ................ 000014a0: 0200 0200 0200 0200 0100 0200 0200 0100 ................ 000014b0: 0100 0000 0100 0100 0100 0000 0100 0100 ................ -000014c0: 0100 0100 045e 8d0d 1400 0000 0000 0000 .....^.......... +000014c0: 0100 0100 f4b7 cd09 1400 0000 0000 0000 ................ 000014d0: 1d00 0000 0000 0000 0100 0100 1500 0000 ................ 000014e0: 1000 0000 0000 0000 630d 0500 0000 0200 ........c....... 000014f0: 1000 0000 0000 0000 f099 0d00 1700 0000 ................ @@ -97548,14 +97548,14 @@ 0017d0b0: b259 0600 8559 0600 5b59 0600 2c59 0600 .Y...Y..[Y..,Y.. 0017d0c0: e158 0600 9058 0600 6058 0600 2858 0600 .X...X..`X..(X.. 0017d0d0: f457 0600 2f74 6d70 2f67 6f6d 6f62 696c .W../tmp/gomobil -0017d0e0: 652d 776f 726b 2d36 3238 3136 3038 3333 e-work-628160833 +0017d0e0: 652d 776f 726b 2d32 3034 3030 3234 3432 e-work-204002442 0017d0f0: 2f73 7263 2f67 6f62 696e 642f 676f 5f6d /src/gobind/go_m 0017d100: 6169 6e2e 676f 0000 2f74 6d70 2f67 6f6d ain.go../tmp/gom -0017d110: 6f62 696c 652d 776f 726b 2d36 3238 3136 obile-work-62816 -0017d120: 3038 3333 2f73 7263 2f67 6f62 696e 642f 0833/src/gobind/ +0017d110: 6f62 696c 652d 776f 726b 2d32 3034 3030 obile-work-20400 +0017d120: 3234 3432 2f73 7263 2f67 6f62 696e 642f 2442/src/gobind/ 0017d130: 7365 715f 616e 6472 6f69 642e 676f 0000 seq_android.go.. 0017d140: 2f74 6d70 2f67 6f6d 6f62 696c 652d 776f /tmp/gomobile-wo -0017d150: 726b 2d36 3238 3136 3038 3333 2f73 7263 rk-628160833/src +0017d150: 726b 2d32 3034 3030 3234 3432 2f73 7263 rk-204002442/src 0017d160: 2f67 6f62 696e 642f 7365 712e 676f 0000 /gobind/seq.go.. 0017d170: 2f76 6172 2f74 6d70 2f64 6973 742f 676f /var/tmp/dist/go 0017d180: 7061 7468 2f73 7263 2f67 6f6c 616e 672e path/src/golang. @@ -113192,14 +113192,14 @@ 001ba270: 0001 0101 0100 0000 0100 005f 6367 6f5f ..........._cgo_ 001ba280: 676f 7479 7065 732e 676f 0000 0000 2f74 gotypes.go..../t 001ba290: 6d70 2f67 6f6d 6f62 696c 652d 776f 726b mp/gomobile-work -001ba2a0: 2d36 3238 3136 3038 3333 2f73 7263 2f67 -628160833/src/g +001ba2a0: 2d32 3034 3030 3234 3432 2f73 7263 2f67 -204002442/src/g 001ba2b0: 6f62 696e 642f 7365 712e 676f 0000 0000 obind/seq.go.... 001ba2c0: 2f74 6d70 2f67 6f6d 6f62 696c 652d 776f /tmp/gomobile-wo -001ba2d0: 726b 2d36 3238 3136 3038 3333 2f73 7263 rk-628160833/src +001ba2d0: 726b 2d32 3034 3030 3234 3432 2f73 7263 rk-204002442/src 001ba2e0: 2f67 6f62 696e 642f 7365 715f 616e 6472 /gobind/seq_andr 001ba2f0: 6f69 642e 676f 0000 0000 2f74 6d70 2f67 oid.go..../tmp/g -001ba300: 6f6d 6f62 696c 652d 776f 726b 2d36 3238 omobile-work-628 -001ba310: 3136 3038 3333 2f73 7263 2f67 6f62 696e 160833/src/gobin +001ba300: 6f6d 6f62 696c 652d 776f 726b 2d32 3034 omobile-work-204 +001ba310: 3030 3234 3432 2f73 7263 2f67 6f62 696e 002442/src/gobin 001ba320: 642f 676f 5f6d 6169 6e2e 676f 0000 0000 d/go_main.go.... 001ba330: 3c61 7574 6f67 656e 6572 6174 6564 3e00 <autogenerated>. 001ba340: 0000 0000 0005 0280 cf0b 0004 0103 f000 ................ @@ -113234,7 +113234,7 @@ 001ba510: 0101 5801 0000 0200 c200 0000 0101 fb0e ..X............. 001ba520: 0d00 0101 0101 0000 0001 0000 012f 746d ............./tm 001ba530: 702f 676f 6d6f 6269 6c65 2d77 6f72 6b2d p/gomobile-work- -001ba540: 3632 3831 3630 3833 332f 7372 632f 676f 628160833/src/go +001ba540: 3230 3430 3032 3434 322f 7372 632f 676f 204002442/src/go 001ba550: 6269 6e64 002f 7661 722f 746d 702f 6469 bind./var/tmp/di 001ba560: 7374 2f67 6f70 6174 682f 706b 672f 676f st/gopath/pkg/go 001ba570: 6d6f 6269 6c65 2f6e 646b 2d74 6f6f 6c63 mobile/ndk-toolc @@ -113260,8 +113260,8 @@ 001ba6b0: 6f6f 6c63 6861 696e 732f 6172 6d2f 6269 oolchains/arm/bi 001ba6c0: 6e2f 2e2e 2f73 7973 726f 6f74 2f75 7372 n/../sysroot/usr 001ba6d0: 2f69 6e63 6c75 6465 002f 746d 702f 676f /include./tmp/go -001ba6e0: 6d6f 6269 6c65 2d77 6f72 6b2d 3632 3831 mobile-work-6281 -001ba6f0: 3630 3833 332f 7372 632f 676f 6269 6e64 60833/src/gobind +001ba6e0: 6d6f 6269 6c65 2d77 6f72 6b2d 3230 3430 mobile-work-2040 +001ba6f0: 3032 3434 322f 7372 632f 676f 6269 6e64 02442/src/gobind 001ba700: 0000 6367 6f2d 6763 632d 7072 6f6c 6f67 ..cgo-gcc-prolog 001ba710: 0000 0000 7374 6469 6e74 2e68 0001 0000 ....stdint.h.... 001ba720: 7365 715f 616e 6472 6f69 642e 6800 0200 seq_android.h... @@ -113293,8 +113293,8 @@ 001ba8c0: 2d74 6f6f 6c63 6861 696e 732f 6172 6d2f -toolchains/arm/ 001ba8d0: 6269 6e2f 2e2e 2f73 7973 726f 6f74 2f75 bin/../sysroot/u 001ba8e0: 7372 2f69 6e63 6c75 6465 002f 746d 702f sr/include./tmp/ -001ba8f0: 676f 6d6f 6269 6c65 2d77 6f72 6b2d 3632 gomobile-work-62 -001ba900: 3831 3630 3833 332f 7372 632f 676f 6269 8160833/src/gobi +001ba8f0: 676f 6d6f 6269 6c65 2d77 6f72 6b2d 3230 gomobile-work-20 +001ba900: 3430 3032 3434 322f 7372 632f 676f 6269 4002442/src/gobi 001ba910: 6e64 002f 7661 722f 746d 702f 6469 7374 nd./var/tmp/dist 001ba920: 2f67 6f70 6174 682f 706b 672f 676f 6d6f /gopath/pkg/gomo 001ba930: 6269 6c65 2f6e 646b 2d74 6f6f 6c63 6861 bile/ndk-toolcha @@ -113471,8 +113471,8 @@ 001bb3e0: 6f6c 6368 6169 6e73 2f61 726d 2f62 696e olchains/arm/bin 001bb3f0: 2f2e 2e2f 7379 7372 6f6f 742f 7573 722f /../sysroot/usr/ 001bb400: 696e 636c 7564 6500 2f74 6d70 2f67 6f6d include./tmp/gom -001bb410: 6f62 696c 652d 776f 726b 2d36 3238 3136 obile-work-62816 -001bb420: 3038 3333 2f73 7263 2f67 6f62 696e 6400 0833/src/gobind. +001bb410: 6f62 696c 652d 776f 726b 2d32 3034 3030 obile-work-20400 +001bb420: 3234 3432 2f73 7263 2f67 6f62 696e 6400 2442/src/gobind. 001bb430: 2f76 6172 2f74 6d70 2f64 6973 742f 676f /var/tmp/dist/go 001bb440: 7061 7468 2f70 6b67 2f67 6f6d 6f62 696c path/pkg/gomobil 001bb450: 652f 6e64 6b2d 746f 6f6c 6368 6169 6e73 e/ndk-toolchains @@ -183567,11 +183567,11 @@ 002cd0e0: 2e63 676f 322e 6300 5f63 676f 5f38 3938 .cgo2.c._cgo_898 002cd0f0: 6432 6164 3234 6165 315f 4366 756e 635f d2ad24ae1_Cfunc_ 002cd100: 6672 6565 002f 746d 702f 676f 6d6f 6269 free./tmp/gomobi -002cd110: 6c65 2d77 6f72 6b2d 3632 3831 3630 3833 le-work-62816083 -002cd120: 332f 7372 632f 676f 6269 6e64 2f73 6571 3/src/gobind/seq +002cd110: 6c65 2d77 6f72 6b2d 3230 3430 3032 3434 le-work-20400244 +002cd120: 322f 7372 632f 676f 6269 6e64 2f73 6571 2/src/gobind/seq 002cd130: 5f61 6e64 726f 6964 2e63 002f 746d 702f _android.c./tmp/ -002cd140: 676f 6d6f 6269 6c65 2d77 6f72 6b2d 3632 gomobile-work-62 -002cd150: 3831 3630 3833 332f 7372 632f 676f 6269 8160833/src/gobi +002cd140: 676f 6d6f 6269 6c65 2d77 6f72 6b2d 3230 gomobile-work-20 +002cd150: 3430 3032 3434 322f 7372 632f 676f 6269 4002442/src/gobi 002cd160: 6e64 006a 766d 0072 6573 6572 7665 6430 nd.jvm.reserved0 002cd170: 0072 6573 6572 7665 6431 0072 6573 6572 .reserved1.reser 002cd180: 7665 6432 0044 6573 7472 6f79 4a61 7661 ved2.DestroyJava @@ -183961,15 +183961,15 @@ 002ce980: 6600 6a6f 626a 0063 6c61 7a7a 0072 6566 f.jobj.clazz.ref 002ce990: 5f63 6c61 7373 0073 6967 006d 006f 626a _class.sig.m.obj 002ce9a0: 0069 7369 6e73 7400 2f74 6d70 2f67 6f6d .isinst./tmp/gom -002ce9b0: 6f62 696c 652d 776f 726b 2d36 3238 3136 obile-work-62816 -002ce9c0: 3038 3333 2f73 7263 2f67 6f62 696e 642f 0833/src/gobind/ +002ce9b0: 6f62 696c 652d 776f 726b 2d32 3034 3030 obile-work-20400 +002ce9c0: 3234 3432 2f73 7263 2f67 6f62 696e 642f 2442/src/gobind/ 002ce9d0: 736e 6f77 666c 616b 6563 6c69 656e 745f snowflakeclient_ 002ce9e0: 616e 6472 6f69 642e 6300 4a61 7661 5f73 android.c.Java_s 002ce9f0: 6e6f 7766 6c61 6b65 636c 6965 6e74 5f53 nowflakeclient_S 002cea00: 6e6f 7766 6c61 6b65 636c 6965 6e74 5f5f nowflakeclient__ 002cea10: 3169 6e69 7400 5f75 6e75 7365 6400 2f74 1init._unused./t 002cea20: 6d70 2f67 6f6d 6f62 696c 652d 776f 726b mp/gomobile-work -002cea30: 2d36 3238 3136 3038 3333 2f73 7263 2f67 -628160833/src/g +002cea30: 2d32 3034 3030 3234 3432 2f73 7263 2f67 -204002442/src/g 002cea40: 6f62 696e 642f 756e 6976 6572 7365 5f61 obind/universe_a 002cea50: 6e64 726f 6964 2e63 0070 726f 7879 5f63 ndroid.c.proxy_c 002cea60: 6c61 7373 5f5f 6572 726f 7200 7072 6f78 lass__error.proxNot sure about
-000014c0: 0100 0100 045e 8d0d 1400 0000 0000 0000 .....^.......... +000014c0: 0100 0100 f4b7 cd09 1400 0000 0000 0000 ................, though. But I guess we'll see how it looks once we ruled the easy issue out.
Trac:
Status: needs_review to needs_revision
Keywords: TorBrowserTeam201906R deleted, TorBrowserTeam201906 added Replying to gk:
They are not but it seems we are close. Just
libgojni.sodiffers for me with what seems to be mostly different tmp directories being used during build:These are reminiscent of the mismatches in comment:4:ticket:22831. There, we dealt with it by running a sed command to clobber random paths of the form /tmp/go-buildXXXXXXXXX and /tmp/go-link-XXXXXXXXX. In this case, it looks like /tmp/gomobile-work-XXXXXXXXX is needed as well.
Not sure about the other one either, though. It doesn't look like a timestamp.
-
There is a new patch for review in branch
bug_28672_v7: https://gitweb.torproject.org/user/boklm/tor-browser-build.git/commit/?h=bug_28672_v7&id=082b4fd8759ef9e88317940d77d091291d0363dfThis gomobile patch is fixing the issue with
/tmp/gomobile-work-XXXXXXXXXpaths: https://gitweb.torproject.org/user/boklm/tor-browser-build.git/tree/projects/gomobile/0001-If-the-GOMOBILE_TMPDIR-env-variable-is-defined-use-t.patch?h=bug_28672_v7&id=082b4fd8759ef9e88317940d77d091291d0363dfHowever there was still an issue with a
/tmp/go-buildXXXXXXXXX. I tried fixing it with ased -i -E -e 's#(/tmp/go-build|/tmp/go-link-)[None..None](../compare/None...None){9}/#\1XXXXXXXXX/#g' libgojni.so, but this did not fix the issue completely as there was still differences after that:--- out/snowflake/snowflake-d11e55aabe37-android-armv7-d0aca3.aar +++ /home/boklm/reproduce-build/tor-browser-build/out/snowflake/snowflake-d11e55aabe37-android-armv7-d0aca3.aar ├── zipinfo {} │ @@ -1,10 +1,10 @@ │ -Zip file size: 1090707 bytes, number of entries: 8 │ +Zip file size: 1090705 bytes, number of entries: 8 │ -rw------- 3.0 unx 154 t- defN 19-Apr-16 16:59 AndroidManifest.xml │ -rw------- 3.0 unx 0 b- stor 19-Apr-16 16:59 R.txt │ -rw------- 3.0 unx 11138 b- defN 19-Apr-16 16:59 classes.jar │ drwx------ 3.0 unx 0 b- stor 19-Apr-16 16:59 jni/ │ drwx------ 3.0 unx 0 b- stor 19-Apr-16 16:59 jni/armeabi-v7a/ │ -rw------- 3.0 unx 3133888 b- defN 19-Apr-16 16:59 jni/armeabi-v7a/libgojni.so │ -rw------- 3.0 unx 25 t- stor 19-Apr-16 16:59 proguard.txt │ drwx------ 3.0 unx 0 b- stor 19-Apr-16 16:59 res/ │ -8 files, 3145205 bytes uncompressed, 1089881 bytes compressed: 65.4% │ +8 files, 3145205 bytes uncompressed, 1089879 bytes compressed: 65.4% ├── jni/armeabi-v7a/libgojni.so │┄ No file format specific differences found inside, yet data differs (ELF 32-bit LSB shared object, ARM, EABI5 version 1 (SYSV), dynamically linked, not stripped) │ @@ -20,16 +20,16 @@ │ 0000130: 0400 0000 52e5 7464 f089 0d00 f099 0d00 ....R.td........ │ 0000140: f099 0d00 1066 0a00 1066 0a00 0600 0000 .....f...f...... │ 0000150: 0800 0000 0400 0000 5300 0000 0400 0000 ........S....... │ 0000160: 476f 0000 3966 3676 6e5f 5755 5876 6950 Go..9f6vn_WUXviP │ 0000170: 6c7a 4a38 6c30 6a47 2f62 5769 6354 4c66 lzJ8l0jG/bWicTLf │ 0000180: 3656 6572 7637 456d 536e 5945 792f 5877 6Verv7EmSnYEy/Xw │ 0000190: 766f 4d72 5359 5250 6a78 345f 3264 4f6f voMrSYRPjx4_2dOo │ -00001a0: 505a 2f35 5767 4c57 5934 555a 4c4a 6e65 PZ/5WgLWY4UZLJne │ -00001b0: 4d53 4146 6f68 6900 0000 0000 0000 0000 MSAFohi......... │ +00001a0: 505a 2f45 7054 3844 3964 6734 765f 6134 PZ/EpT8D9dg4v_a4 │ +00001b0: 6648 4c51 7767 3700 0000 0000 0000 0000 fHLQwg7......... │ 00001c0: 0000 0000 0000 0000 0100 0000 0000 0000 ................ │ 00001d0: 0000 0000 1200 0000 4300 0000 0000 0000 ........C....... │ 00001e0: 0000 0000 1200 0000 5000 0000 48d9 0b00 ........P...H... │ 00001f0: 4400 0000 1200 0b00 7b00 0000 9cd9 0b00 D.......{....... │ 0000200: 0800 0000 1200 0b00 9800 0000 8cd9 0b00 ................ │ 0000210: 0800 0000 1200 0b00 bf00 0000 94d9 0b00 ................ │ 0000220: 0800 0000 1200 0b00 e600 0000 00d9 0b00 ................ │ @@ -326,15 +326,15 @@ │ 0001450: 0100 0100 0100 0100 0100 0100 0100 0100 ................ │ 0001460: 0100 0100 0100 0100 0100 0200 0200 0100 ................ │ 0001470: 0100 0100 0100 0100 0000 0200 0100 0200 ................ │ 0001480: 0200 0200 0100 0100 0100 0200 0200 0200 ................ │ 0001490: 0200 0200 0200 0200 0200 0200 0100 0100 ................ │ 00014a0: 0200 0200 0200 0200 0100 0200 0200 0100 ................ │ 00014b0: 0100 0000 0100 0100 0100 0000 0100 0100 ................ │ -00014c0: 0100 0100 04e7 8403 1400 0000 0000 0000 ................ │ +00014c0: 0100 0100 24e0 0308 1400 0000 0000 0000 ....$........... │ 00014d0: 1d00 0000 0000 0000 0100 0100 1500 0000 ................ │ 00014e0: 1000 0000 0000 0000 630d 0500 0000 0200 ........c....... │ 00014f0: 1000 0000 0000 0000 f099 0d00 1700 0000 ................ │ 0001500: f426 0f00 1700 0000 f826 0f00 1700 0000 .&.......&...... │ 0001510: fc26 0f00 1700 0000 0027 0f00 1700 0000 .&.......'...... │ 0001520: 0427 0f00 1700 0000 0827 0f00 1700 0000 .'.......'...... │ 0001530: 0c27 0f00 1700 0000 1027 0f00 1700 0000 .'.......'......So I added added a patch to go, to make it use a fixed tmp directory, if the directory does not exist yet, which is fixing the issue: https://gitweb.torproject.org/user/boklm/tor-browser-build.git/tree/projects/go/0001-Use-fixed-go-build-tmp-directory.patch?h=bug_28672_v7&id=082b4fd8759ef9e88317940d77d091291d0363df
Trac:
Status: needs_revision to needs_review
Keywords: TorBrowserTeam201906 deleted, TorBrowserTeam201906R added -
Thanks! It seems I got reproducible builds now, nice! Here come some review notes:
93b7f904ec931c56adf9f84b50756cf2f7776f38 -- okay 524f9d9b6cfc6418e237fcd6c5264bb301a2411f -- okay 093ab72ddc37be1f4932213c4a78ff1e55348500 -- not okay;
- So, we need a particular OpenSSL version or is the one in Stretch just too old? Could you add a comment here explaining what happens (and ideally pointing to the problematic code)?
-
+ target_os = ["win", "android"]-- we don't target Windows right now and should leave that target out - What's the reason for moving the
gclient configpart outside of theif [ ! -d "src"]one, in particular as this affects non-Android platforms as well. And what does it mean? That it is not run for Linux and macOS anymore? - We could change the
GN_ARGS-adding parts to
[% IF c("var/linux") -%] GN_ARGS+= [% ELSIF c("var/osx") -%] GN_ARGS+= [% ELSIF c("var/android") -%] GN_ARGS+= [% END -%]commit 082b4fd8759ef9e88317940d77d091291d0363df -- not okay; Just copying the .aar over does not mean it gets included into the final result. I think we'd need to update the
android-dependencies.patchfile for that as well. However, I am not exactly sure about the way forward here as the .aar approach seems to be in line with https://github.com/guardianproject/AndroidPluggableTransports but that's not how we currently utilize PTs on mobile which is having a binary like we do on desktop. I'll bring this as a topic for the meeting discussion up today.Trac:
Status: needs_review to needs_revision
Keywords: TorBrowserTeam201906R deleted, TorBrowserTeam201907 added -
Replying to gk:
- What's the reason for moving the
gclient configpart outside of theif [ ! -d "src"]one, in particular as this affects non-Android platforms as well. And what does it mean? That it is not run for Linux and macOS anymore?
If the
gclient configis behind theif [ ! -d "src"], then changes to gclient config are not taken into account unless we remove thesrcdirectory. So moving it outside allows us to update this config without removing thesrcdirectory.It seems we should have a patch moving that part outside of the
if [ ! -d "src"]on the stable branch too, otherwise building master will update the gclient config, and then building stable will keep using the master version of the gclient config.The
target_osline means fetching dependencies specific for those platforms. It seems we don't need specific dependencies for the Linux and macOS builds as we had notarget_osdefined before.With this change we will fetch Android dependencies (and Windows ones, but we can remove it for now), and will include it in the webrtc tarball used for every platforms. This means we include some dependencies not needed in the Linux and macOS builds, but this avoids generating one tarball for each platform.
- What's the reason for moving the
-
Replying to boklm:
Replying to gk:
- What's the reason for moving the
gclient configpart outside of theif [ ! -d "src"]one, in particular as this affects non-Android platforms as well. And what does it mean? That it is not run for Linux and macOS anymore?
If the
gclient configis behind theif [ ! -d "src"], then changes to gclient config are not taken into account unless we remove thesrcdirectory. So moving it outside allows us to update this config without removing thesrcdirectory.It seems we should have a patch moving that part outside of the
if [ ! -d "src"]on the stable branch too, otherwise building master will update the gclient config, and then building stable will keep using the master version of the gclient config.Okay, sounds reasonable. Yes, we should keep that in mind once we need this on stable (which we currently don't do).
The
target_osline means fetching dependencies specific for those platforms. It seems we don't need specific dependencies for the Linux and macOS builds as we had notarget_osdefined before.With this change we will fetch Android dependencies (and Windows ones, but we can remove it for now), and will include it in the webrtc tarball used for every platforms. This means we include some dependencies not needed in the Linux and macOS builds, but this avoids generating one tarball for each platform.
Yes, that seemed to be perfectly reasonable. Just the Windows part is not needed yet (maybe not even ever given the build issues), thus let's remove it for now.
- What's the reason for moving the
-
FWIW: the fix for legacy/trac#25483 (moved) just landed and we should redo our Android patches based on that.
Got a build working here: https://gitweb.torproject.org/user/cohosh/tor-browser-build.git/log/?h=snowflake_android
I'm doing another build with legacy/trac#34043 (moved) and then testing that on my phone to see what it's like.
-
Hmm okay it seems that this ticket is just for building snowflake and legacy/trac#30318 (moved) is for integrating with Tor Browser. I'll put this in needs_review then since reproducible builds are now working.
Here's the single patch needed: https://gitweb.torproject.org/user/cohosh/tor-browser-build.git/commit/?h=snowflake_android&id=bec6cbace3d6c5125beac926645341b0921fb792
Trac:
Status: assigned to needs_review
Component: Circumvention/Snowflake to Applications/Tor Browser -
Okay here's an additional commit on the snowflake_android branch that bundles snowflake with tor browser for android. I followed the same procedure used for obfs4 to enable it only for alpha and nightlies in
rbm.confand bundle it withtor-onion-proxy-library.Trac:
Status: needs_revision to needs_review -
Replying to cohosh:
Okay here's an additional commit on the snowflake_android branch that bundles snowflake with tor browser for android. I followed the same procedure used for obfs4 to enable it only for alpha and nightlies in
rbm.confand bundle it withtor-onion-proxy-library.So, it turns out that Pluto is actually not bundling
snowflakeat all. I guess you could just copysnowflake-clientover in the for loop already dealing withobfs4proxy. I am not sure why we actually copy things over to all the different arch dirs given that we build per arch, but that's up for a different bug.Please squash the commits and rebase onto latest
master, we are close with this ticket (modulo testing that things actually work) I think. Thanks.Trac:
Status: needs_review to needs_revision -
So, it turns out that Pluto is actually not bundling snowflake at all. I guess you could just copy snowflake-client over in the for loop already dealing with obfs4proxy. I am not sure why we actually copy things over to all the different arch dirs given that we build per arch, but that's up for a different bug.
I was going to ask what Pluto is and how it knows about obfs4. Does it need to build snowflake?
Please squash the commits and rebase onto latest master, we are close with this ticket (modulo testing that things actually work) I think. Thanks.
Here it is: https://gitweb.torproject.org/user/cohosh/tor-browser-build.git/log/?h=snowflake_android
I started my own build of this rebased branch.
Trac:
Status: needs_revision to needs_review -
Replying to cohosh:
So, it turns out that Pluto is actually not bundling snowflake at all. I guess you could just copy snowflake-client over in the for loop already dealing with obfs4proxy. I am not sure why we actually copy things over to all the different arch dirs given that we build per arch, but that's up for a different bug.
I was going to ask what Pluto is and how it knows about obfs4. Does it need to build snowflake?
Pluto is the Guardian Project's pluggable transport library which is deprecated. We should redo at some point the whole tor-android-service/tor-onion-proxy-library parts which use Pluto but for now we just play along. :)
Please squash the commits and rebase onto latest master, we are close with this ticket (modulo testing that things actually work) I think. Thanks.
Here it is: https://gitweb.torproject.org/user/cohosh/tor-browser-build.git/log/?h=snowflake_android
I started my own build of this rebased branch.
Thanks. Let's just use one
forloop maybe (see my previous comment), like so (untested):# Extract obfs4proxy from TorBrowser/Tor/PluggableTransports/obfs4proxy tar --strip-components=4 -xf $rootdir/[% c('input_files_by_name/obfs4') %] # Extract snowflake from TorBrowser/Tor/PluggableTransports/snowflake tar --strip-components=4 -xf $rootdir/[% c('input_files_by_name/snowflake') %] # Overwrite the obfs4proxy binary provided by Pluto and add snowflake for d in external/pluto/bin/*; do cp obfs4proxy $d/ cp snowflake-client $d/ done rm obfs4proxy rm snowflake-client? I don't think we need two separate
forloops for what we want to do.I'll try to come up with some patches for legacy/trac#30318 (moved) over the weekend (I copy and pasted things for the browser part already I believe), so we can actually test on mobile.
Trac:
Status: needs_review to needs_revision
Keywords: TorBrowserTeam202005R deleted, TorBrowserTeam202005 added -
Woo okay here's a working patch with those fixes, thanks: https://gitweb.torproject.org/user/cohosh/tor-browser-build.git/commit/?h=snowflake_android&id=5ef430adf7b7232566955aba86695ea88fd3b642
Replying to cohosh:
Woo okay here's a working patch with those fixes, thanks: https://gitweb.torproject.org/user/cohosh/tor-browser-build.git/commit/?h=snowflake_android&id=5ef430adf7b7232566955aba86695ea88fd3b642
Thanks! I rebased this and merged it with commit
853e75e9276432a39153db2e3fa54375af9b8822onmaster.Trac:
Status: needs_review to closed
Resolution: N/A to fixed
Keywords: N/A deleted, tbb-10.0a1 addedmentioned in issue legacy/trac#30318 (moved)
mentioned in issue legacy/trac#31010 (moved)
