README.HACKING 12.6 KB
Newer Older
boklm's avatar
boklm committed
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
Hacking on Tor Browser Build
============================

This file tries to list the main things to know when making changes to
the Tor Browser build.


rbm documentation
-----------------

If you go to directory rbm/doc, you can find the rbm documentation. You
can build it with 'make html' or 'make man'.


Using and defining options
--------------------------

Options can be used in templates with the following syntax:

  [% c("option_name") %]

More details about the templates syntax can be found in
rbm/doc/rbm_templates.7 and [http://template-toolkit.org/docs/manual/index.html].

Some options have a specific meaning to rbm. You can see their descriptions
in rbm/doc/rbm_config.7 and rbm/doc/rbm_input_files.7.

When the option does not have a specific meaning to rbm, it is a good
idea to define it under var/ to avoid potential conflicts in option names
with a future version of rbm.

The options can be defined in different places:

- rbm.conf for options available to all components
- project/$project/config for options available to one component
- rbm.local.conf for user defined options

In each of those places, an option can be defined:

- at the root of the document
- under targets/$target/, in which case the definition only applies when
  a specific target is defined

The targets are usually used to select:

- the platform: torbrowser-linux-x86_64, torbrowser-linux-i686,
47
  torbrowser-windows-i686, torbrowser-windows-x86_64, torbrowser-osx-x86_64,
48
49
  torbrowser-android-armv7, torbrowser-android-aarch64, torbrowser-android-x86,
  torbrowser-android-x86_64
boklm's avatar
boklm committed
50
51
52
- the channel: release, nightly, alpha

The targets torbrowser-linux-x86_64, torbrowser-linux-i686,
53
torbrowser-windows-i686, torbrowser-windows-x86_64, torbrowser-osx-x86_64,
54
55
56
57
58
59
torbrowser-android-armv7, torbrowser-android-x86, torbrowser-android-aarch64,
torbrowser-android-x86_64 are special cases. They do not contain options
directly, instead they contain a list of other targets. For instance, the
torbrowser-linux-x86_64 target is pointing to the linux-x86_64 and linux
targets. You should define an option under the linux target if it applies to
Linux on both architectures, or under the linux-x86_64 if it only applies to
60
the x86_64 architecture.
boklm's avatar
boklm committed
61

62
An option that is defined at the root of rbm.conf can be overridden by
boklm's avatar
boklm committed
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
an other definition under a target, or inside projects/$project/config.
You can find the complete priority order in rbm/doc/rbm_config.7.


Defining a project's filename
-----------------------------

The filename option defines the output file or directory from the build
of a component. If the file or directory exists in the out/$component
directory, the component will not be rebuilt when it is used as a
dependency from an other component.

The filename is usually something like this:

  filename: '[% project %]-[% c("version") %]-[% c("var/osname") %]-[% c("var/build_id") %].tar.gz'

The var/build_id value is a hash that is based on the build inputs of
the component:
 - the build script (after template evaluation)
 - the container definition
 - the input files (the filename when it is a dependency on an other
   project, the filename and hash of its content otherwise)
This means that each time the build script, the container or one of the
dependencies is modified, the output filename will change and a rebuild
of the component will be required.

The version and var/osname values could be removed from the filename,
but they are useful to get an idea of what a file contains.


Adding some Linux/Windows/OSX specific changes to a build script
----------------------------------------------------------------

You can use the following template syntax in the build scripts:

  [% IF c("var/linux") -%]
    # do something for linux
  [% ELSIF c("var/windows") -%]
    # do something for windows
  [% ELSIF c("var/osx") -%]
    # do something for osx
  [% END -%]

You can also use var/linux-x86_64 and var/linux-i686 for things that
only apply to x86_64 and i686 linux builds. You can use the var/release,
var/alpha and var/nightly options to do things depending on the channel.

As an alternative you can define an option with a different value
depending on the target:

  targets:
    linux:
      var:
        do_something: 'do something for linux'
    windows-i686:
      var:
        do_something: 'do something for windows'
    osx-x86_64:
      var:
        do_something: 'do something for osx'

And in the build script, use:

  [% c("var/do_something") %]


Evaluating a component's build script
-------------------------------------

If you want to look at the build script used to build a component for a
specific platform/channel, you can use the following command:

   $ ./rbm/rbm showconf $component build --target $channel --target $platform

$component should be replaced by the name of the component.

$channel should be one of the following:

 - release

 - alpha

 - nightly

$platform should be one of the following:

 - torbrowser-linux-x86_64

 - torbrowser-linux-i686

 - torbrowser-windows-i686

155
156
 - torbrowser-windows-x86_64

boklm's avatar
boklm committed
157
158
 - torbrowser-osx-x86_64

159
160
 - torbrowser-android-armv7

161
162
 - torbrowser-android-aarch64

163
164
 - torbrowser-android-x86

165
166
 - torbrowser-android-x86_64

167
For example, to see tor's build script for linux x86_64 on the alpha
boklm's avatar
boklm committed
168
169
170
171
172
173
174
175
channel, you can use:

  $ ./rbm/rbm showconf tor build --target alpha --target \
    torbrowser-linux-x86_64

If the component you are looking at has many dependencies, the display
can take some time as various build_id values need to be computed. If
you don't care about the accuracy of input and output file names, you
176
can add '--target no_build_id' to the command line. For instance, if you
boklm's avatar
boklm committed
177
178
179
180
181
182
183
184
want to look at the build script for the tor-browser component (which
has a lot of dependencies), you can use:

  $ ./rbm/rbm showconf tor build --target alpha --target \
     torbrowser-linux-x86_64 --target no_build_id

The same type of commands can be used to look at any option values,
replacing build with the name of the option you want to look at. For
185
instance, if you want to know the output filename of tor on linux-x86_64
boklm's avatar
boklm committed
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
on the alpha channel, you can use:

  $ ./rbm/rbm showconf tor filename --target alpha --target \
    torbrowser-linux-x86_64


Building a single component
---------------------------

When you are working on some changes to a component, before creating a
full bundle with those changes, you might want to try a build of the
component you modified only.

This can be done with the following command:

   $ ./rbm/rbm build $component --target $channel --target $platform

See the previous section "Evaluating a component's build script" for a
list of possible values for $channel and $platform.

206
For instance, if you want to build tor for linux-x86_64 on the alpha
boklm's avatar
boklm committed
207
208
209
210
211
212
213
214
channel, you can run:

   $ ./rbm/rbm build tor --target alpha --target torbrowser-linux-x86_64

To find the resulting file from the build, you can use 'ls -ltr out/tor/'
to find the file with the last modification time.


215
216
217
Patching Firefox (or an other component)
----------------------------------------

218
If you want to test a Firefox patch, the easiest way to do it is to
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
copy the patch file to the projects/firefox/ directory, then edit
projects/firefox/config to add the new patch to the list of input_files:

  - filename: patch-for-XXXX.patch

Then edit projects/firefox/build to add a line somewhere (probably just
before running the configure script) to apply the patch:

  patch -p1 < $rootdir/patch-for-XXXX.patch

You can now run 'make testbuild' (or an other build target) to start a
build with the patch.

As an alternative, if you have your patch in a git repository, you can
edit projects/firefox/config to change the git_url option to point to
your git repository, and change the git_hash option to point to the
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
commit you want to build. If the new git_hash option is not pointing to
a signed tag, you will also need to comment the 'tag_gpg_id: 1' line.
The git_hash option can point to a tag, a commit, a branch or anything
that is understood by git.

If you specify a branch in the git_hash option, you don't need to prefix
it with a git remote name as all branches from the git repository
selected in the git_url option are created as local branches.

If you want to work on a local git repository, it is not a good idea to
work on the git_clones/firefox repository directly. Instead you should
create a separate git repository and update the git_url option with a
file:// URL pointing to your repository.

After changing the git_url option or when new commits have been added
to a branch, you should run "make fetch" before starting a build to get
the new commits. If you want to fetch new commits automatically when
starting a new build you can add a "fetch: 1" line to
projects/firefox/config.
254

255
256
257
258
Remember that the git_hash option has different definitions for alpha
and nightly builds, so you should modify the right one depending on
what type of build you are doing (see also the "Types of builds" section).

259
260
261
262
263
264
265
266
267
268
The first alternative is not working in case patches contain binary diffs,
which the patch command can't handle. However, working with submodules can
make the git branch alternative tricky to use. In that case you can resort
to a third option by installing git in the build container (via var/deps in
the respective config file). Then you can change to the submodule directory
in the build script and do a

  git apply patch-for-XXXX.patch

after adding the patch to the project's config file (see above).
269

270
271
272
273
274
275
276
277
278
The Firefox mozconfig files
---------------------------

In the gitian build, we are using the mozconfig files included in the
tor-browser.git repository (the .mozconfig, .mozconfig-mac and
.mozconfig-mingw files).

In the rbm build however, we need to make some small modifications to
those files, so we are instead using mozconfig files stored in the
279
projects/firefox/ directory, ignoring the .mozconfig files from the
280
281
tor-browser.git repository.

282
This could change in the future, when we are not using Gitian anymore.
283
284


boklm's avatar
boklm committed
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
Debugging a build error
-----------------------

If the build fails, a debugging shell will automatically be opened in
the container used for the build (unless you set debug to 0 in
rbm.local.conf).

Before using the debugging shell, you can first look at the build logs
in an other terminal (the filename of the log file is indicated in the
rbm output).

In the debug shell, you can type 'bash' to get a shell with completion.
You can then look at the 'build' file, and copy paste some of its commands
to debug the problem. If you need to edit files, vim is not installed
in the build containers, but vi can be used instead.

When you press ctrl+d (twice if you opened a bash shell) the debug shell
is closed and the container containing the failed build is removed.

304
305
The path to the container should be printed on the screen in case you
want to backup its rootfs to be able to look at it later.
boklm's avatar
boklm committed
306

307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323

Testing an rbm patch
--------------------

When you are working on a patch to rbm, you might want to try a Tor
Browser build using your patched version of rbm. You could patch the
rbm in the rbm/ directory, however your patch can be reverted if you
use any of the makefile rules that does a 'git submodule update'.
To avoid this you can clone the rbm git repository to a separate
directory, where you will apply your patch. To do a build using your
patched rbm, take the command from the makefile, but replace $(rbm)
with the path to your patched rbm.

For example, if you want to try a Linux x86_64 alpha build, you can run:
  $ /path_to_rbm/rbm build release --target alpha --target \
                                                torbrowser-linux-x86_64

324

325
326
Types of builds: nightly, alpha, release, and testbuild
-------------------------------------------------------
327
328
329

The testbuild makefile target allows you to do a build quickly in the
testbuild directory, skipping the generation of all the locales and the
330
331
332
333
334
335
336
337
338
.mar files. This is useful during development.

In the case of Android builds, we are generating a multi-locale apk,
contrary to the desktop builds where we have one bundle for each locale.
Removing locales in a multi-locale bundle does not make a significant
difference in build time, therefore we still include all the locales in
the Android testbuild. There are also no .mar files generated in the
Android builds, so currently, in the Android case, the only difference
introduced by the testbuild target is the output directory.
339
340
341
342
343
344
345
346
347

By default the testbuild is based on the alpha build. All the options
can have a different definition for the alpha, release and nightly builds.
Usually the git_hash option has a different definition for the nightly
builds in order to point to the master branch.

If you want your testbuild target to create builds based on nightly
rather than alpha, you can edit your rbm.local.conf file and uncomment
the targets/torbrowser-testbuild definition.