Consider mounting out as a bind
In tpo/tpa/team#40964 (closed) micah suggested we switch to bind mounts to avoid copying.
I agree that it'd be a great idea, especially since almost all of the files we copy are archives we then just extract and don't use.
At least theoretically, I don't have numbers at the moment about the overhead that copies create, but I'd love if we instrumented RBM to profile the steps that take a long time.
E.g., starting to build firefox-android
on my machine takes 8 mins from the creation of the log to the start of the build script.
In tb-build-05, the same takes 11 minutes (I don't expect file copies to be the main cause of this overhead, though).
(But these are exceptions, most of the projects are below 30 seconds; see #40061 (closed)).
I see a couple of concerns, but we could easily solve them in my opinion:
- we want to keep visible only a limited set of inputs
- We could share a temporary directory inside as
out
on which we hardlink the actual inputs. AFAIK, hardlinking them to the directory of the container doesn't work in some environments, e.g., if we use a ramdisk (like we do intb-build-06
), but binding volume should work.
- We could share a temporary directory inside as
- it might be more difficult to find the exact file
- maybe we could change
c("input_files_by_name/...")
to be absolute paths. - if we prepare a single directory outside the container and expose that, finding files should be easy.
- maybe we could change