Build our own cctools for macOS cross-compilation

Ray Donnelly has been working on merging his OSX cross compiler patches into the crosstools-ng project, which should allow us to more easily build his compilers from source instead of relying on a binary blob.

Jotting down Ray's instructions in case anyone wants to try this:

On Linux it is possible now to build the equivalent of what toolchain4 provided (for OSX only - iOS still has some build issues)

You should be able to clone: https://github.com/diorcety/crosstool-ng.git

.. checkout the cctools-llvm branch, and enter:

./configure --prefix=<SOMEWHERE>/ct-ng-build
make
make install
export PATH="${PATH}":<SOMEWHERE>/ct-ng-build
mkdir <SOMEWHEREELSE>/ct-ng-final
cd <SOMEWHEREELSE>/ct-ng-final
ct-ng list-samples
ct-ng i686-apple-darwin11 (or i686-apple-darwin10)

Check the differences between samples/i686-apple-darwin11 and samples/i686-apple-darwin10 too, the main one being that 10 copies the sysroot into the final build whereas 11 doesn't. This means you need to pass --sysroot to the compilers built with 11 but in theory don't need to for the 10 ones (of course 10 is set up to use MacOSX10.6.sdk while 11 is set up to use MacOSX10.7.sdk).

We've been adding clang to the project most recently which has caused us to disabled llvm-gcc for now and the addition of clang isn't ready yet (it is nearly there but I am going on holiday for 2 weeks after this week so I'm not sure if we will get it finished before then - likely not).

So, all the above instructions are still valid; go with commit 7d555f284b6977e64640a30bcec77597580d3049 if you can. Any problems give me a shout. FWIW, llvm/clang is only now coming on-line for Darwin software anyway, and vanilla gcc-4.2 (i.e. not even llvm-gcc-4.2) is much more reliable for building OSX software. Seems Apple focussed more on iOS than OSX for clang.

Let me know how it goes. We've started to submit some patches upstream to crosstool-ng this last week, but I have a feeling it could be a drawn out process. We will regularly merge the other way the meantime though.

added GeorgKoppen201805 in Legacy / Trac TorBrowserTeam201805R in Legacy / Trac component::applications/tor browser in Legacy / Trac owner::tbb-team in Legacy / Trac parent::24632 in Legacy / Trac priority::very high in Legacy / Trac resolution::fixed in Legacy / Trac severity::normal in Legacy / Trac status::closed in Legacy / Trac tbb-rbm in Legacy / Trac type::task in Legacy / Trac labels

Trac:
Cc: N/A to g.koppen@jondos.de

Mike, one question: Is there a reason for using i686-apple-darwin11 in the current setup when compiling and not i686-apple-darwin10 given that we want to support OSX10.6 as the minVersion (and are therefore using the 10.6 SDK anyway)?

Trac:
Status: new to needs_information

Oh, and while I am at it: Could you send me Ray's email address? I already encountered bugs while trying to compile the toolchain that seem to be fun bugs :)

Mike, are you using llvmgcc or normal gcc from toolchain4 at present? If normal gcc, then I'd recommend that you guys build from the latest of the cctools-llvm branch to get the latest fixes (llvmgcc is disabled at present, IMHO it was never very reliable, even the official Apple binaries). If you update to the latest then clang can be built now, however this is very untested so far and if you look in the TODO file in the root folder you'll get a better picture of some things that may still be problematic with clang.

Georg, there's no reason to use darwin11 instead of darwin10. When I started this work it was with a darwin11 SDK so it's purely a matter of habit for me. The merge to crosstool-ng is being done by Yann Diorcet and myself; I handle the darwin11 samples and he does the same for the darwin10 ones.

A compiler built with the darwin11 SDK (MacOSX10.7.sdk) can be used fine to build software for darwin10 using -mmacosx-version-min=10.5, However feel free to base your version on i686-apple-darwin10 (e.g. flosoft's MacOSX10.6.sdk) and the i686-apple-darwin10 sample instead.

I recommend studying the two crosstool.config files:

samples/i686-apple-darwin10/crosstool.config and samples/i686-apple-darwin11/crosstool.config

.. these are just samples of course. You could make your own, e.g. darwin-gitian. The final compiler prefixes don't depend on the sample folder name but rather the options specified in the crosstool.config file itself.

Some options that should be noted are:

CT_DARWIN_SDK_PATH="${HOME}/MacOSX10.7.sdk" .. this is where you placed your OSX SDK.

CT_DARWIN_COPY_SDK_TO_SYSROOT=n .. setting this to y means that some headers and dylibs from the OSX SDK are copied into the installation prefix. Doing this means you do not need to pass -sysroot to the compiler on the commandline, with the down-side being that you'd not be able to distribute the built toolchain due to copyrights and the OSX SDK license.

CT_LLVM_V_3_3=y .. specifies the version of LLVM (and clang) to use. LLVM is compiled into cctools for LTO support and also, obviously for clang. Toolchain4 used LLVM 2.7 which is quite old at this stage. 3.3 is the latest release. I'm more interested in modernising things than using old versions (though reliability trumps all other concerns).

CT_DEBUGGABLE_TOOLCHAIN=n .. expect to use 20GB if you set this to y. The built tools (and also those installed) are built at -O0 -ggdb and left un-stripped.

I think it would be sensible for me to setup exactly the same environment you are using. Is this Ubuntu? If so, can you point me to the exact ISO and any scripts you use to prepare it for development? I will then make a new VM.

If you've got some logs (build.log) detailing the "fun bugs" please post them.

Trac:
Username: mingwandroid

Something else I should mention is that the old GCC in the crosstool-ng port doesn't yet support the -arch and -Xarch flags (whereas toolchain4 does).

This is because those flags are actually handled by driverdriver.c which is an Apple-specific program that's compiled outside of the normal GCC/llvmgcc compilation and when run calls the real GCC once for each arch specified (using the standard -m32 or -m64 flag). It then calls lipo to merge the objects for each arch together into a fat object/binary.

I'm not sure how the tbb build system for Mac invokes gcc; I would hope it is easy to change it from using "-arch i386" to "-m32" and/or "-arch x86_64" to "-m64", of course, if you build fat binaries in a single invocation then this is a bigger problem.

Having said that, adding the driverdriver program is high priority and something I expect to do tomorrow.

Finally, clang 'natively' supports these flags but I'd be (pleasantly) surprised if switching tbb to build with clang was trivial.

Trac:
Username: mingwandroid

Replying to mingwandroid:

Mike, are you using llvmgcc or normal gcc from toolchain4 at present? If normal gcc, then I'd recommend that you guys build from the latest of the cctools-llvm branch to get the latest fixes (llvmgcc is disabled at present, IMHO it was never very reliable, even the official Apple binaries).

Normal gcc and yes, I am using the latest cctools-llvm branch.

Georg, there's no reason to use darwin11 instead of darwin10. When I started this work it was with a darwin11 SDK so it's purely a matter of habit for me. The merge to crosstool-ng is being done by Yann Diorcet and myself; I handle the darwin11 samples and he does the same for the darwin10 ones.

A compiler built with the darwin11 SDK (MacOSX10.7.sdk) can be used fine to build software for darwin10 using -mmacosx-version-min=10.5, However feel free to base your version on i686-apple-darwin10 (e.g. flosoft's MacOSX10.6.sdk) and the i686-apple-darwin10 sample instead.

That was actually the reason why I asked. We use already flosoft's MacOSX10.6.sdk and I hoped to avoid using the 10.7 SDK to get a compiler to use that one with the 10.6 SDK... Good.

I recommend studying the two crosstool.config files:

samples/i686-apple-darwin10/crosstool.config and samples/i686-apple-darwin11/crosstool.config

.. these are just samples of course. You could make your own, e.g. darwin-gitian. The final compiler prefixes don't depend on the sample folder name but rather the options specified in the crosstool.config file itself.

Thanks I'll think about doing an own while I digest that whole system.

I think it would be sensible for me to setup exactly the same environment you are using. Is this Ubuntu? If so, can you point me to the exact ISO and any scripts you use to prepare it for development? I will then make a new VM.

The gitian build system is working with (at least) the current LTS (12.04). I order to build the Mac TBB you do the following

git clone https://git.torproject.org/builders/tor-browser-bundle.git
cd tor-browser-bundle/gitian
make prep

Install all the stuff you are asked to and run

make prep

again until you have all the necessary sources (i.e. until |make prep| does not give you any errors anymore) Then do

./mkbundle-mac.sh

and it should give you (after a while) Mac TBBs in a new directory (currently 3.0-alpha-4). Anyway, if you have issues with setting it up. Just write me a mail and we'll sort these things out.

If you've got some logs (build.log) detailing the "fun bugs" please post them.

So I have found so far three issues (disclaimer: I am not sure yet if there is anything you can do about as this might actually be things belonging to the source packages used in building the compiler):

1. I hit intermittent GMP configure failures due to newly created files being older than distributed ones. (log: cross_mac_gmp1) That is likely due to the gitian build environment but it would be good to somehow avoid that. I am wondering whether the reason here is some old version of GMP? I have been building mingw-w64 from source for a while with a similar gitian environment but never hit this problem.

2. 'groff' was missing (the check in the configure script is there but although 'groff' is needed for building the script is not returning with an error if it is not available) (cross_mac_build1). Installing the necessary package probably solves the issue.

3. a Makefile.config(?) is missing? (cross_mac_build2).

Now, what bothers me most, though, is that Firefox 24 is not working with a gcc4.2 anymore (at least Mozilla is saying that in the configure script and I don't have a reason to doubt that). I am under the impression that your cross-compilers are (by default) gcc4.2, no? If so, what are our options here? Are there non-interactive ways to change the gcc version?

Trac:
cross_mac_gmp1

Trac:
cross_mac_build1

Trac:
cross_mac_build2

Replying to gk:

The gitian build system is working with (at least) the current LTS (12.04). I order to build the Mac TBB you do the following {{{ git clone https://git.torproject.org/builders/tor-browser-bundle.git cd tor-browser-bundle/gitian make prep }}} Install all the stuff you are asked to and run {{{ make prep }}} again until you have all the necessary sources (i.e. until |make prep| does not give you any errors anymore) Then do {{{ ./mkbundle-mac.sh }}} and it should give you (after a while) Mac TBBs in a new directory (currently 3.0-alpha-4). Anyway, if you have issues with setting it up. Just write me a mail and we'll sort these things out.

I forgot to add that the relevant files for cross-compiling the TBB code are gitian-tor.yml and gitian-firefox.yml in tor-browser-bundle/gitian/descriptors/mac. This should give you a hint on how your cross-compilers are currently used in the gitian build setup. If you have questions about that as well, drop me a mail.

Replying to gk:

Replying to mingwandroid:

Mike, are you using llvmgcc or normal gcc from toolchain4 at present? If normal gcc, then I'd recommend that you guys build from the latest of the cctools-llvm branch to get the latest fixes (llvmgcc is disabled at present, IMHO it was never very reliable, even the official Apple binaries).

Normal gcc and yes, I am using the latest cctools-llvm branch.

Georg, there's no reason to use darwin11 instead of darwin10. When I started this work it was with a darwin11 SDK so it's purely a matter of habit for me. The merge to crosstool-ng is being done by Yann Diorcet and myself; I handle the darwin11 samples and he does the same for the darwin10 ones.

A compiler built with the darwin11 SDK (MacOSX10.7.sdk) can be used fine to build software for darwin10 using -mmacosx-version-min=10.5, However feel free to base your version on i686-apple-darwin10 (e.g. flosoft's MacOSX10.6.sdk) and the i686-apple-darwin10 sample instead.

That was actually the reason why I asked. We use already flosoft's MacOSX10.6.sdk and I hoped to avoid using the 10.7 SDK to get a compiler to use that one with the 10.6 SDK... Good.

I recommend studying the two crosstool.config files:

samples/i686-apple-darwin10/crosstool.config and samples/i686-apple-darwin11/crosstool.config

.. these are just samples of course. You could make your own, e.g. darwin-gitian. The final compiler prefixes don't depend on the sample folder name but rather the options specified in the crosstool.config file itself.

Thanks I'll think about doing an own while I digest that whole system.

I think it would be sensible for me to setup exactly the same environment you are using. Is this Ubuntu? If so, can you point me to the exact ISO and any scripts you use to prepare it for development? I will then make a new VM.

The gitian build system is working with (at least) the current LTS (12.04). I order to build the Mac TBB you do the following

Should I use the 32bit version of the LTS?

{{{ git clone https://git.torproject.org/builders/tor-browser-bundle.git cd tor-browser-bundle/gitian make prep }}} Install all the stuff you are asked to and run {{{ make prep }}} again until you have all the necessary sources (i.e. until |make prep| does not give you any errors anymore) Then do {{{ ./mkbundle-mac.sh }}} and it should give you (after a while) Mac TBBs in a new directory (currently 3.0-alpha-4). Anyway, if you have issues with setting it up. Just write me a mail and we'll sort these things out.

If you've got some logs (build.log) detailing the "fun bugs" please post them.

So I have found so far three issues (disclaimer: I am not sure yet if there is anything you can do about as this might actually be things belonging to the source packages used in building the compiler):

1. I hit intermittent GMP configure failures due to newly created files being older than distributed ones. (log: cross_mac_gmp1) That is likely due to the gitian build environment but it would be good to somehow avoid that. I am wondering whether the reason here is some old version of GMP? I have been building mingw-w64 from source for a while with a similar gitian environment but never hit this problem.

In general, how does gitian handle filestamp dependencies? I'm under the impression that it uses .so injection to force specific time functions to return the same thing, given that, I'm a little confused. Is there a mechanism to force some fixed negative delta onto files created by specific tools (tar and patch come to mind)?

2. 'groff' was missing (the check in the configure script is there but although 'groff' is needed for building the script is not returning with an error if it is not available) (cross_mac_build1). Installing the necessary package probably solves the issue.

Ok this sounds like an upstream crosstool-ng problem. I will put it on the TODO list for now.

3. a Makefile.config(?) is missing? (cross_mac_build2).

Now, what bothers me most, though, is that Firefox 24 is not working with a gcc4.2 anymore (at least Mozilla is saying that in the configure script and I don't have a reason to doubt that). I am under the impression that your cross-compilers are (by default) gcc4.2, no? If so, what are our options here? Are there non-interactive ways to change the gcc version?

4.2.1 was as the last GCC that Apple provided. They heavily patched this version (and made llvmgcc from it) and they never updated beyond that (understandable given the level and nature of patching, of course they could've opted to make their changes cleanly and worked with GCC core developers to upstream but I suspect GCC was something they worked on grudgingly). They switched from GCC to clang, so when you say that Firefox 24 is not working with gcc4.2 anymore, I can only assume that for Mac, they switched to building with clang - or maybe llvmgcc, if you can determine the exact details of this it would be helpful, also can you find out it requires libc++? When do you plan to switch over to Firefox 24?

I will download your git repo, give it a test and study your logs at lunchtime. You didn't say whether you use the -arch flag or not, but no problem I can take a look.

Trac:
Username: mingwandroid

Just for reference: Even though the build (i686-apple-darwin10) is much smoother outside the gitian setup it still does not go through on a clean Ubuntu VM, see output in cross_mac_vm.

Trac:
cross_mac_vm

Trac:
Username: mingwandroid

build.log.tar.xz

Ray's build.log (tar.xz'ed due to size limits)

Trac:
Username: mingwandroid

Trac:
Username: mingwandroid

build-i686.log.tar.xz

Ray's build.log (tar.xz'ed due to size limits - targeting OS X, not iPhone!)

Trac:
Username: mingwandroid

I added two logs, the first is actually targeting ARM (iPhone), so you probably should ignore that one for now.

Was your log complete? In mine, "[INFO ] Installing cctools for host" doesn't happen until line 42686 whereas it's the first line in your log. I'm guessing you snipped your log to avoid size limits and so that it can be referred to via URLs? I think having your complete log would be helpful if this is the case.

From your log: "[CFG ] configure: WARNING: llvm-config (/home/firefox/x-tools/i686-apple-darwin10/bin/i686-apple-darwin10-llvm-config) not found, will use LLVM 2.7 defaults" is interesting. It indicates that LLVM wasn't built or installed correctly.

Trac:
Username: mingwandroid

Replying to mingwandroid:

I added two logs, the first is actually targeting ARM (iPhone), so you probably should ignore that one for now.

Was your log complete? In mine, "[INFO ] Installing cctools for host" doesn't happen until line 42686 whereas it's the first line in your log. I'm guessing you snipped your log to avoid size limits and so that it can be referred to via URLs? I think having your complete log would be helpful if this is the case.

From your log: "[CFG ] configure: WARNING: llvm-config (/home/firefox/x-tools/i686-apple-darwin10/bin/i686-apple-darwin10-llvm-config) not found, will use LLVM 2.7 defaults" is interesting. It indicates that LLVM wasn't built or installed correctly.

Attached is the full build log (cross_mac_vm_full.tar.bz2).

Trac:
cross_mac_vm_full.tar.bz2

Replying to mingwandroid: [snip]

Should I use the 32bit version of the LTS?

That does not matter as gitian is creating own VMs for building.

[snip]

In general, how does gitian handle filestamp dependencies? I'm under the impression that it uses .so injection to force specific time functions to return the same thing, given that, I'm a little confused. Is there a mechanism to force some fixed negative delta onto files created by specific tools (tar and patch come to mind)?

Mike, could you comment on that. I am not sure about the details...

Now, what bothers me most, though, is that Firefox 24 is not working with a gcc4.2 anymore (at least Mozilla is saying that in the configure script and I don't have a reason to doubt that). I am under the impression that your cross-compilers are (by default) gcc4.2, no? If so, what are our options here? Are there non-interactive ways to change the gcc version?

4.2.1 was as the last GCC that Apple provided. They heavily patched this version (and made llvmgcc from it) and they never updated beyond that (understandable given the level and nature of patching, of course they could've opted to make their changes cleanly and worked with GCC core developers to upstream but I suspect GCC was something they worked on grudgingly). They switched from GCC to clang, so when you say that Firefox 24 is not working with gcc4.2 anymore, I can only assume that for Mac, they switched to building with clang

Yes, they did: https://bugzilla.mozilla.org/show_bug.cgi?id=733905

• or maybe llvmgcc, if you can determine the exact details of this it would be helpful, also can you find out it requires libc++?

Will see what I can find.

When do you plan to switch over to Firefox 24?

As soon as possible I think. TBB with Firefox 24 should be ready if the ESR 17 reaches its EOL (which is IIRC in about 7-8 weeks).

I will download your git repo, give it a test and study your logs at lunchtime. You didn't say whether you use the -arch flag or not, but no problem I can take a look.

We use no -arch flags if I see it correctly.

I've had an initial look and here's what I've gleamed:

There's currently an issue which forces us to need to clean out both the builddir and any existing installation before doing subsequent builds in the same directories. For you this would mean doing:

rm -rf /home/firefox/crosstool-ng/ct-ng-final /home/firefox/x-tools

You're currently basing your build on the darwin10 sample, but actually you'd be better off starting from my darwin11 one (see note [1]) so I'd replace samples/i686-apple-darwin10/crosstool.config with samples/i686-apple-darwin11/crosstool.config and then, to turn it back into a darwin10 config, change the following entries:

CT_DARWIN_MAC_OSX_V_10_7=y to: CT_DARWIN_MAC_OSX_V_10_6=y

CT_DARWIN_VERSION="11" to: CT_DARWIN_VERSION="10"

CT_DARWIN_SDK_PATH="${HOME}/MacOSX10.7.sdk" to: CT_DARWIN_SDK_PATH="${HOME}/MacOSX10.6.sdk"

.. if you want to be able to use the generated toolchains without passing --sysroot $MacOSXSDKDir then change (I don't recommend this - see note [2]) change

CT_DARWIN_COPY_SDK_TO_SYSROOT=n to: CT_DARWIN_COPY_SDK_TO_SYSROOT=y

.. depending on whether you would want to debug issues if/when we run into them change (likely worthwhile for now, provided you've got ~20GB to spare)

CT_DEBUGGABLE_TOOLCHAIN=n to: CT_DEBUGGABLE_TOOLCHAIN=y

Notes:

[1] I suspect I maintain my darwin11 samples a bit more regularly, whereas Yann generates new ones each time via make menuconfig. Also my samples will always use the latest versions of things that are in a reasonably working state (LLVM and clang 3.3 in this case)

[2] I don't recommend this as it's not a use-case I have verified as working, specifically, AFAIK the frameworks are not copied over so building anything that requires more components than a C and C++ library is likely to fail. Further, toolchains built like this cannot be legally distributed.

Trac:
Username: mingwandroid

Trac:
cross_mac_build3.tar.bz2

Replying to mingwandroid:

You're currently basing your build on the darwin10 sample, but actually you'd be better off starting from my darwin11 one (see note [1]) so I'd replace samples/i686-apple-darwin10/crosstool.config with samples/i686-apple-darwin11/crosstool.config and then, to turn it back into a darwin10 config, change the following entries:

I did that but am running in the same error as in cross_mac_build2 (the Makefile.config issue mentioned above). I attached the whole log (cross_mac_build3.tar.bz2) in case it helps.

Are you running it via gitian now? The reason I ask is that the unpacking stage of the log is full of:

[FILE ] gmp-5.1.1/gen-fac.c [FILE ] tar: gen-fac.c: time stamp 2013-02-11 15:29:14 is 413911753.757075149 s in the future

.. clearly 2013-02-11 is in the past (given a sane, non-US date ordering at least)

Then the LLVM build itself has lots of:

[ALL ] make[4]: Warning: File `/home/ubuntu/build/crosstool-ng/ct-ng-final/.build/i686-apple-darwin10/build/build-LLVM-host-i686-build_pc-linux-gnu/Makefile.llvmbuild' has modification time 0.81 s in the future [ALL ] llvm[4]: Regenerating /home/ubuntu/build/crosstool-ng/ct-ng-final/.build/i686-apple-darwin10/build/build-LLVM-host-i686-build_pc-linux-gnu/Makefile.config

.. so it's constantly re-generating the Makefiles and doing other "maintainer mode" build steps that shouldn't need to be done with a released tarball.

Trac:
Username: mingwandroid

Replying to mingwandroid:

Are you running it via gitian now?

Yes. This bug is about getting everything running in gitian which is currently my main concern (or better my main concern is getting Fx 24 ESR cross-compiled for Mac which is actually legacy/trac#9829 (closed)). I am happy to test with a clean VM without gitian, though, (to debug other/related issues) but that needs to be done in a different bug (not sure where crosstools-ng has its bugtracker).

Replying to gk:

Replying to mingwandroid:

Are you running it via gitian now?

Yes. This bug is about getting everything running in gitian which is currently my main concern (or better my main concern is getting Fx 24 ESR cross-compiled for Mac which is actually legacy/trac#9829 (closed)). I am happy to test with a clean VM without gitian, though, (to debug other/related issues) but that needs to be done in a different bug (not sure where crosstools-ng has its bugtracker).

I agree that gitian is the end result here, however having crosstool-ng working outside of gitian is a necessary first step toward that result. If you feel this step merits a ticket of its own then that's fine, please go ahead and create one.

Trac:
Username: mingwandroid

I am happy if the bug report lives on this tracker, but https://github.com/diorcety/crosstool-ng/issues would arguably be a better place for it. I can try to create one there if you prefer?

Trac:
Username: mingwandroid

Finally, before creating this bug report it's definitely worth knowing first if there's a failure outside of gitian ;-)

Trac:
Username: mingwandroid

Replying to mingwandroid:

I am happy if the bug report lives on this tracker, but https://github.com/diorcety/crosstool-ng/issues would arguably be a better place for it. I can try to create one there if you prefer?

As soon as my build machine is ready again I'll try your suggestions in a clean VM outside of gitian and report an issue in the crosstool-ng bugtracker (if there are any, that is).

I used:

http://www.google.co.uk/ig/directory?type=gadgets&url=echo3.net/dcalc/gadget.xml

.. and entered "now - 413911773 seconds" and it said:

"Monday, 28-Aug-2000"

gnutar will extract files with the date stamp as embedded in the archives; having these dates 13 years in the future is likely to cause the build failure you are seeing. I think you need to fix your date at "the-date-of-the-most-recent-tarball-that-you-use + 1 second" or something like that. Mike's input on this would be very useful.

Of course if you've got solutions to any of these issues that require options being added to crosstool-ng then I can try to make those changes and get them merged when everything is working.

Trac:
Username: mingwandroid

Okay, in case it helps:

Gitian relies on libfaketime to set the clock to a fixed value to deal with embedded timestamps in archives and in the build process.

See: https://blog.torproject.org/blog/deterministic-builds-part-two-technical-details and see: https://github.com/wolfcw/libfaketime/blob/master/README for how libfaketime works (basically via LD_PRELOAD; and we are setting FAKETIME to "2000-01-01 00:00:00")

That said, even if there is a lot of regenerating going on I currently don't see why Makefile.config should be missing because of this. Thus, I am not sure whether dealing with the faketime issue is sufficient to solve this particular problem.

Replying to gk:

Okay, in case it helps:

Gitian relies on libfaketime to set the clock to a fixed value to deal with embedded timestamps in archives and in the build process.
}}}
See: https://blog.torproject.org/blog/deterministic-builds-part-two-technical-details and see: https://github.com/wolfcw/libfaketime/blob/master/README for how libfaketime works (basically via LD_PRELOAD; and we are setting FAKETIME to "2000-01-01 00:00:00")

Forgot to add that we do {{{ find -type f | xargs touch --date="$REFERENCE_DATETIME"

in source directories where "$REFERENCE_DATETIME" is currently "2000-01-01 00:00:00". That might be a thing worth trying regarding the crosstools-ng sources...

Replying to gk:

Okay, in case it helps: {{{ Gitian relies on libfaketime to set the clock to a fixed value to deal with embedded timestamps in archives and in the build process. }}} See: https://blog.torproject.org/blog/deterministic-builds-part-two-technical-details and see: https://github.com/wolfcw/libfaketime/blob/master/README for how libfaketime works (basically via LD_PRELOAD; and we are setting FAKETIME to "2000-01-01 00:00:00")

That said, even if there is a lot of regenerating going on I currently don't see why Makefile.config should be missing because of this. Thus, I am not sure whether dealing with the faketime issue is sufficient to solve this particular problem.

There's continual regeneration of Makefile.config in the presence of recursive make (-j5). So I can easily imagine that at various points Makefile.config does not exist. Removing and re-writing a file is not a multithread-safe atomic operation.

Trac:
Username: mingwandroid

I added an option for this feature: From menuconfig's perspective it is called "Date and time to set on all source files" From config file perspective it's called CT_SRC_REFERENCE_DATETIME. I also added it to samples/i686-apple-darwin11/crosstool.config as: CT_SRC_REFERENCE_DATETIME="1999-12-31 23:59:59" It would be good if you could pull the latest, try this out and see where things stand.

Trac:
Username: mingwandroid

Oh, I set the timestamp on directories not just files in-case any Makefile uses a directory as a dependency and also because it looks more tidy.

Trac:
Username: mingwandroid

Okay. The build failed. See cross_mac_build4.tar.bzip for the full log. Gitian gave me (additionally it seems) the following messages:

[00:00] / 
[INFO ]    Forcing date and time of patched sources to 1999-12-31 23:59:59
[00:00] / touch: cannot touch `space/test.h': Not a directory
touch: cannot touch `space/test.h.result': Not a directory
touch: cannot touch `space/test2.m.in': Not a directory
touch: cannot touch `space/test1.m.in.result': Not a directory
touch: cannot touch `space/test1.m.in': Not a directory
touch: cannot touch `space/test2.m.in.result': Not a directory
[ERROR]  
[00:00] / 
[ERROR]  >>
[00:00] / 
[ERROR]  >>  Build failed in step 'Forcing date and time of patched sources to 1999-12-31 23:59:59'
[00:00] / 
[ERROR]  >>        called in step 'Extracting and patching toolchain components'
[00:00] / 
[ERROR]  >>        called in step '(top-level)'
[00:00] / 
[ERROR]  >>
[00:00] / 
[ERROR]  >>  Error happened in: main[scripts/crosstool-NG.sh@651]

Trac:
cross_mac_build4.tar.bz2

Ah, I had locally disabled clang from my config, and it the "find | xargs touch" command doesn't correctly handle spaces in paths (which one of the clang test-suite files contains).

I pushed a fix for this now using:

find .. -exec touch --date= .. "{}" ;

.. instead.

Trac:
Username: mingwandroid

Looks good. Now, the gitian build process is stuck with https://github.com/diorcety/crosstool-ng/issues/1 as well.

This was down to a bash-ism that crept into the cctools patch-set. It should be fixed now. Please update and re-test.

Trac:
Username: mingwandroid

I made a test set of binaries (outside of gitian though) on a clean Ubuntu 12.04.3 VM: https://www.dropbox.com/s/k5s09xhqv3mv2ti/i686-apple-darwin10.tar.xz

.. the exact config I used was: https://www.dropbox.com/s/q9z66w119fvhwd2/crosstool.config

I'm now giving gitian a go. First thing I noticed is that: sudo torsocks apt-get install

.. is very noisy:

07:54:57 libtorsocks(933): The symbol res_query() was not found in any shared library. The error reported was: not found! 07:54:57 libtorsocks(933): The symbol res_search() was not found in any shared library. The error reported was: not found! 07:54:57 libtorsocks(933): The symbol __res_send() was not found in any shared library. The error reported was: not found! 07:54:57 libtorsocks(933): The symbol res_querydomain() was not found in any shared library. The error reported was: not found!

.. should I be concerned about this?

Also, you've been trying to build crosstool-ng under gitian, is that just a question of prepending "sudo torsocks" in front of all the crosstool-ng build commands? If you have a new script that you can commit then that would be good.

Trac:
Username: mingwandroid

Replying to mingwandroid:

I made a test set of binaries (outside of gitian though) on a clean Ubuntu 12.04.3 VM: https://www.dropbox.com/s/k5s09xhqv3mv2ti/i686-apple-darwin10.tar.xz

.. the exact config I used was: https://www.dropbox.com/s/q9z66w119fvhwd2/crosstool.config

I'm now giving gitian a go. First thing I noticed is that: sudo torsocks apt-get install

.. is very noisy:

07:54:57 libtorsocks(933): The symbol res_query() was not found in any shared library. The error reported was: not found! 07:54:57 libtorsocks(933): The symbol res_search() was not found in any shared library. The error reported was: not found! 07:54:57 libtorsocks(933): The symbol __res_send() was not found in any shared library. The error reported was: not found! 07:54:57 libtorsocks(933): The symbol res_querydomain() was not found in any shared library. The error reported was: not found!

.. should I be concerned about this?

No.

Also, you've been trying to build crosstool-ng under gitian, is that just a question of prepending "sudo torsocks" in front of all the crosstool-ng build commands?

No. 'torsocks' is just used to download the dependencies via Tor.

If you have a new script that you can commit then that would be good.

Let me know when you are ready (i.e. the VMs are up and you built a Mac bundle successfully). I'll send you a patch which you can apply on top of your local branch and which should give you the environment I am currently using.

That said, I've been running now into three different issues which are likely to due to the gitian build process: 1) the linker got killed 2) not enough space on the VM 3) another build failure for yet unknown reasons. I fixed 1) and 2) and am currently debugging 3).

But your latest two commits fixing bashisms seemed to work. I'll test that on my Ubuntu VM again as soon as my build machine is not occupied with gitian build stuff and will comment on the github bugtracker then.

Can I run the VM from within a VM?

It was last night when I last tried but I remember getting something about needing to do:

export USE_LXC=1

.. because AFAIR it said VM-in-a-VM wouldn't work so to do this instead? Is that the case? Are there some settings I can apply to make VM-in-a-VM work?

Trac:
Username: mingwandroid

Good questions. But, honestly, I don't know the answers to them. I never tried that setup as building without nested VMs is already slow enough for me :)

You probably hit the KVM check telling you something like

Most likely, this means you will need to use LXC.

Please run this in your shell before each build:
 export USE_LXC=1

LXC mode is currently kind of highly experimental and presumably broken on trunk. Your better bet is getting KVM running inside your VM.

It seems the VM gitian creates by default is not big enough, sigh. Trying a larger one...

CT_DEBUGGABLE_TOOLCHAIN=n results in a much reduced space requirement.

Trac:
Username: mingwandroid

Replying to mingwandroid:

CT_DEBUGGABLE_TOOLCHAIN=n results in a much reduced space requirement.

Yes, that is how I fixed 2) in comment 32 but it was not enough to solve 3)...