Bug 1661434 - added debugging and taskcluster pages from MDN to in-tree. r=sylvestre

Debugging A Hang On OS X

========================

This article contains historical information about older versions of OS X.

See `How to Report a Hung

Firefox <https://developer.mozilla.org/en-US/docs/Mozilla/How_to_report_a_hung_Firefox>`__ instead.

If you find a hang in an application, it is very useful for the

developer to know where in the code this hang happens, especially if he

or she can't reproduce it. Below are steps you can use to attach

so-called "Samples" to bug reports.

Creating the sample on Mac OS X 10.5 (XCode < 4.2)

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

#. When the application is still hung, open up Spin Control.app (it’s in

   your <tt>/Developer/Applications/Performance Tools/</tt> folder; if

   it is missing, install the latest `Computer Hardware Understanding

   Development (CHUD)

   Tools <http://developer.apple.com/tools/download/>`__ from Apple).

#. By default, sampling of any hanging application will begin

   automatically.

#. After about 3-4 seconds, select the hanging application in the

   “Detected Hangs” window and click the “Interrupt Sampling” button.

   Note that sampling **will quickly hog up a lot of memory** if you

   leave it on for too long!

#. When it's done parsing the data, click the "Show text report" button;

   a new window will open with a couple of rows with stacktraces for all

   the threads in the sampled application.

#. If you need to upload the sample to Bugzilla, select all the sample

   text, copy it into your favorite text editor, and save as a

   plain-text file.

#. For a more detailed view of the sample data, make sure the sample you

   just recorded is selected in the “Detected Hangs” window and click on

   the “Open…” button.

Creating the sample on Mac OS X 10.4

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

#. When the application is still hung, open up Sampler.app (it’s in your

   <tt>/Developer/Applications/Performance Tools/</tt> folder; if it is

   missing, install the latest `Computer Hardware Understanding

   Development (CHUD)

   Tools <http://developer.apple.com/tools/download/>`__ from Apple.).

#. Choose File > Attach...

#. Select "firefox-bin" or the hung application you want a sample from.

#. Now, click "Start recording". Note that only 3-4 seconds usually

   suffice, note that **this will quickly hog up a lot of memory** if

   you leave it on for too long!

#. When it's done parsing the data, you should now have a couple of rows

   with stacktraces for all the threads in the sampled application.

#. When you have the raw Sample data, you can't just save that and

   attach it to a bug, because the format is not very usable (unless the

   developer is a Mac hacker). Export it using Tools > Generate Report,

   and attach this as a text file to the bug report.

The final data will show where the application spent all its time when

it was hung, and hopefully your bug will be easier to fix now!

See also

~~~~~~~~

`Debugging on Mac OS X <https://developer.mozilla.org/en-US/docs/Mozilla/Debugging/Debugging_on_Mac_OS_X>`__

Debugging On Mac Os X

=====================

This document explains how to debug Mozilla-derived applications such as

Firefox, Thunderbird, and SeaMonkey on macOS using Xcode. If you want to

debug from the terminal see :ref:`Debugging Mozilla with

lldb <Debugging Firefox with LLDB>`. For specific

information on a way to debug hangs, see `Debugging a hang on OS

X <https://developer.mozilla.org/en-US/docs/Mozilla/Debugging/Debugging_a_hang_on_OS_X>`__.

Creating a debuggable build

---------------------------

First, you need to build the application you're going to debug using

``--disable-optimize``\ ``--enable-debug-symbols`` in your .mozconfig

(also add ``--enable-debug`` if you want assertions etc. compiled in).

See :ref:`Building Firefox for macOS <Building Firefox On MacOS>`

if you need help creating your own build.

Debugging Firefox on macOS 10.14+

---------------------------------

macOS 10.14 introduced Notarization and Hardened Runtime features for

improved application security. macOS 10.15 went further, requiring

applications to be Notarized with Hardened Runtime enabled in order to

launch (ignoring workarounds.) When run on earlier macOS versions,

Notarization and Hardened Runtime settings have no effect.

Official Builds

~~~~~~~~~~~~~~~

At this time, official builds of Firefox 69 and later are Notarized.

Uplifts to Firefox 68 and ESR 68 to enable Notarization are planned.

**As a result, it is not possible to attach a debugger to these official

Firefox releases on macOS 10.14+ without disabling System Integrity

Protection (SIP).** This is due to Notarization requiring Hardened

Runtime to be enabled with the ``com.apple.security.get-task-allow``

entitlement disallowed. **Rather than disabling SIP (which has security

implications), it is recommended to debug with try builds or local

builds. The differences are explained below.**

try Server Builds

~~~~~~~~~~~~~~~~~

In most cases, developers needing to debug a build as close as possible

to the production environment should use a `try

build <https://wiki.mozilla.org/ReleaseEngineering/TryServer>`__. These

builds enable Hardened Runtime and only differ from production builds in

that they are not Notarized which should not otherwise affect

functionality, (other than the ability to easily launch the browser on

macOS 10.15+ -- see quarantine note below). At this time, developers can

obtain a Hardened Runtime build with the

``com.apple.security.get-task-allow`` entitlement allowed by submitting

a try build and downloading the dmg generated by the "Rpk" shippable

build job. A debugger can be attached to Firefox processes of these

builds. try builds use the ``developer.entitlements.xml`` file from the

source tree while production builds use ``production.entitlements.xml``.

**On macOS 10.15+, downloaded try builds will not launch by default

because Notarization is required. To workaround this problem, remove the

quarantine extended attribute from the downloaded Nightly:**

  ``$ xattr -r -d com.apple.quarantine /Path/to/Nightly.app``

Local Builds

~~~~~~~~~~~~

Local builds of mozilla-central do not enable Hardened Runtime and hence

do not have debugging restrictions. As a result, some functionality will

be permitted on local builds, but blocked on production builds which

have Hardened Runtime enabled. `Bug

1522409 <https://bugzilla.mozilla.org/show_bug.cgi?id=1522409>`__ was

filed to automate codesigning local builds to enable Hardened Runtime by

default and eliminate this discrepancy.

To obtain a Hardened Runtime build without using try infrastructure, a

developer can manually codesign builds using the macOS ``codesign(1)``

command with the ``developer.entitlements.xml`` file from the tree. This

requires creating a codesigning identity.

Disabling System Integrity Protection (SIP)

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

If debugging a production build is required, follow Apple's documented

steps for disabling System Integrity Protection (SIP). Note that

disabling SIP bypasses Hardened Runtime restrictions which can mask some

bugs that only occur with Hardened Runtime so it is recommended to test

fixes with SIP enabled. **Disabling SIP has system security implications

that should be understood before taking this step.**

Creating an Xcode project

-------------------------

If you try to create a new Xcode project in an existing directory

then Xcode will delete its existing contents (Xcode will warn you

beforehand). To work around that, the steps below have you initialize

the project outside the Mozilla source tree, close the project, copy

the .xcodeproj project "file" into the source tree, and then reopen

the project to finish setting it up.

Note also that since Xcode 7.3.1 it doesn't seem to be possible to

have the Xcode project live outside the source tree. If you try to do

that then Xcode will simply **copy** the source files under the

project directory rather than link to them (still the case in Xcode

10?) which breaks debugging and the possibility to

modify-rebuild-relaunch from inside Xcode.

These steps were last updated for Xcode 10.3:

#. Open Xcode, and create a new Project with File > New Project. Select

   the "Cross-platform" tab then under the "Other" template group select

   the "Empty" project type. the click Next. Name the project and click

   Next. Create/select a temporary directory to contain the project and

   then click Create.

#. Before going any further, close the project (File > Close Project)

   and open Finder. Find the \*.xcodejproj directory in the temporary

   directory, move it into your Mozilla source tree, and then

   double-click on it to reopen it.

#. In the left-hand pane in Xcode you should see a tree item where the

   root item has the project name. If the temporary directory that you

   originally created the Xcode project in is under that, right click it

   and delete it. Now, right click on the root item, select 'Add files

   to "<project-name>"', select all the files and directories in your

   source directory, untick "Copy items if needed", then click Add.

   (These will then be progressively added under the root item

   <project-name> in the left-hand pane. Note that subdirectories may

   initially appear to be empty, but they too will progressively be

   populated as Xcode processes the sourse files. Once done, you should

   be able to open any file quickly by hitting Cmd-Shift-O and typing in

   the name of a file.)

#. In the Product menu, select Scheme > New Scheme and name your scheme

   (for example, "Debug"). After you click OK, Xcode should open the

   settings window for the new scheme. (If not, then open its settings

   from the Product > Edit Scheme menu.)

#. Select "Run" on the left-hand side of the settings window, then

   select the "Info" tab. Set the Executable by clicking on "None" and

   selecting "Other...". A new dialog titled "Choose an executable to

   launch" will pop up. Browse to the ``.app`` file that you want to

   debug (``Firefox.app``, ``Nightly``\ ``Debug.app`` etc). The ``.app``

   file is typically found inside the ``dist`` folder in your build

   directory.

#. If you are debugging Firefox, Thunderbird, or some other application

   that supports multiple profiles, using a separate profile for

   debugging purposes is recommended. See "Having a profile for

   debugging purposes" below. Select the "Arguments" tab in the scheme

   editor, and click the '+' below the "Arguments passed on launch"

   field. Add "-P *profilename*", where *profilename* is the name of a

   profile you created previously. Repeat that to also add the argument

   "-no-remote".

#. Also in the "Arguments" panel, you may want to add an environment

   variable MOZ_DEBUG_CHILD_PROCESS set to the value 1 to help with

   debugging e10s.

#. Select "Build" from the left of the scheme editor window, and check

   that there is nothing listed under Targets (otherwise it may cause

   problems when you try to run the executable for debugging since you

   will get build errors).

#. Click "Close" to close the scheme editor.

At this point you can run the application from Xcode, and when you pause

or hit breakpoints it should show open the correct source file at the

correct line.

Setting up lldb

---------------

``lldb`` is the debugger XCode provides/uses.

.. warning::

   One important issue that the Mozilla .lldbinit file fixes is that by

   default some breakpoints will be listed as "pending", and XCode will

   not stop at them. If you don't include the Mozilla's .lldbinit, you

   must at least put

   ``settings set target.inline-breakpoint-strategy always`` in your

   ``$HOME/.lldbinit`` as recommended on :ref:`Debugging Mozilla with

   lldb <Debugging Firefox with LLDB>`.

The

`.lldbinit <http://dxr.mozilla.org/mozilla-central/source/.lldbinit>`__

file in the source tree imports many useful `Mozilla specific lldb

settings, commands and

formatters <https://dxr.mozilla.org/mozilla-central/source/python/lldbutils/README.txt>`__

into ``lldb``, but you may need to take one of the following steps to

make sure this file is used.

If you are using ``lldb`` on the command line (independently of Xcode)

and you will always run it from either the top source directory, the

object directory or else the dist/bin subdirectory of the object

directory, then adding the following setting to your ``$HOME/.lldbinit``

is sufficient:

::

   settings set target.load-cwd-lldbinit true

*However*, if you will run lldb from a different directory, or if you

will be running it indirectly by debugging in Xcode (Xcode always runs

lldb from "/"), then this setting will not help you. Instead, add the

following to your ``$HOME/.lldbinit``:

::

   # This automatically sources the Mozilla project's .lldbinit as soon as lldb

   # starts or attaches to a Mozilla app (that's in an object directory).

   #

   # This is mainly a workaround for Xcode not providing a way to specify that

   # lldb should be run from a given directory.  (Xcode always runs lldb from "/",

   # regardless of what directory Xcode was started from, and regardless of the

   # value of the "Custom working directory" field in the Scheme's Run options.

   # Therefore setting `settings set target.load-cwd-lldbinit true` can't help us

   # without Xcode providing that functionality.)

   #

   # The following works by setting a one-shot breakpoint to break on a function

   # that we know will both run early (which we want when we start first start the

   # app) and run frequently (which we want so that it will trigger ASAP if we

   # attach to an already running app).  The breakpoint runs some commands to

   # figure out the object directory path from the attached target and then

   # sources the .lldbinit from there.

   #

   # NOTE: This scripts actions take a few seconds to complete, so the custom

   # formatters, commands etc. that are added may not be immediately available.

   #

   breakpoint set --name nsThread::ProcessNextEvent --thread-index 1 --auto-continue true --one-shot true

   breakpoint command add -s python

       # This script that we run does not work if we try to use the global 'lldb'

       # object, since it is out of date at the time that the script runs (for

       # example, `lldb.target.executable.fullpath` is empty).  Therefore we must

       # get the following objects from the 'frame' object.

       target = frame.GetThread().GetProcess().GetTarget()

       debugger = target.GetDebugger()

       # Delete our breakpoint (not actually necessary with `--one-shot true`):

       target.BreakpointDelete(bp_loc.GetBreakpoint().GetID())

       # For completeness, find and delete the dummy breakpoint (the breakpoint

       # lldb creates when it can't initially find the method to set the

       # breakpoint on):

       # BUG WORKAROUND! GetID() on the *dummy* breakpoint appears to be returning

       # the breakpoint index instead of its ID.  We have to add 1 to correct for

       # that! :-(

       dummy_bp_list = lldb.SBBreakpointList(target)

       debugger.GetDummyTarget().FindBreakpointsByName("nsThread::ProcessNextEvent", dummy_bp_list)

       dummy_bp_id = dummy_bp_list.GetBreakpointAtIndex(0).GetID() + 1

       debugger.GetDummyTarget().BreakpointDelete(dummy_bp_id)

       # "source" the Mozilla project .lldbinit:

       os.chdir(target.executable.fullpath.split("/dist/")[0])

       debugger.HandleCommand("command source -s true " + os.path.join(os.getcwd(), ".lldbinit"))

   DONE

see :ref:`Debugging Mozilla with

lldb <Debugging Firefox with LLDB>`. for more information.

Having a profile for debugging purposes

---------------------------------------

It is recommended to create a separate profile to debug with, whatever

your task, so that you don't lose precious data like Bookmarks, saved

passwords, etc. So that you're not bothered with the profile manager

every time you start to debug, expand the "Executables" branch of the

"Groups & Files" list and double click on the Executable you added for

Mozilla. Click the plus icon under the "Arguments" list and type "-P

<profile name>" (e.g. "-P MozillaDebug"). Close the window when you're

done.

Running a debug session

-----------------------

Make sure breakpoints are active (which implies running under the

debugger) by opening the Product menu and selecting "Debug / Activate

Breakpoints" (also shown by the "Breakpoints" button in the top right

section of the main window). Then click the "Run" button or select "Run"

from the Product menu.

Setting breakpoints

~~~~~~~~~~~~~~~~~~~

Setting a breakpoint is easy. Just open the source file you want to

debug in Xcode, and click in the margin to the left of the line of code

where you want to break.

During the debugging session, each time that line is executed, the

debugger will break there, and you will be able to debug it.

.. warning::

   Note that with the default configuration, some breakpoints will be

   listed as "pending", and XCode will not stop at them. If you don't

   include the Mozilla's .lldbinit, you must at least put

   ``settings set target.inline-breakpoint-strategy always`` in your

   ``$HOME/.lldbinit`` as recommended on :ref:`Debugging Mozilla with

   lldb <Debugging Firefox with LLDB>`.

Using Mozilla-specific lldb commands

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

If you included the .lldbinit when `Setting up

lldb <#setting-up-lldb>`__, you can use Mozilla-specific lldb commands

in the console, located in the Debug area of XCode. For example, type

``js`` to see the JavaScript stack. For more information, see :ref:`Debugging

Mozilla with lldb <Debugging Firefox with LLDB>`.

Debugging e10s child processes

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Using XCode to debug child processes created by an e10s-enabled browser

is a little trickier than debugging a single-process browser, but it can

be done. These directions were written using XCode 6.3.1

#. Complete all the steps above under "Creating the Project"

#. From the "Product" menu, ensure the scheme you created is selected

   under "Scheme", then choose "Scheme > Edit Scheme"

#. In the resulting popup, click "Duplicate Scheme"

#. Give the resulting scheme a more descriptive name than "Copy of

   Scheme"

#. Select "Run" on the left-hand side of the settings window, then

   select the "Info" tab. Set the Executable by clicking on the

   "Executable" drop-down, and selecting the ``plugin-container.app``

   that is inside the app bundle of the copy of Firefox you want to

   debug.

#. On the same tab, under "Launch" select "Wait for executable to be

   launched"

#. On the "Arguments" tab, remove all arguments passed on launch.

Now you're ready to start debugging:

#. From the "Product" menu, ensure the scheme you created above is

   selected under "Scheme"

#. Click the "Run" button. The information area at the top of the window

   will show "Waiting for plugin-container to launch"

#. From a command line, run your build of Firefox. When that launches a

   child process (for example, when you start to load a webpage), XCode

   will notice and attach to that child process. You can then debug the

   child process like you would any other process.

#. When you are done debugging, click the "Stop" button and quit the

   instance of Firefox that you were debugging in the normal way.

For some help on using lldb see :ref:`Debugging Mozilla with

lldb <Debugging Firefox with LLDB>`.

Other resources

---------------

Apple has an extensive list of `debugging tips and

techniques <https://developer.apple.com/library/mac/#technotes/tn2124/_index.html>`__.

Questions? Problems?

~~~~~~~~~~~~~~~~~~~~

Try asking in our IRC channels

`#developers <irc://irc.mozilla.org/developers>`__ or

`#macdev <irc://irc.mozilla.org/macdev>`__.

    versioncontrol

    config

    how-tos

    task-graph

    reference

    release-promotion-action

    balrog

    setting-up-an-update-server

    partials

    signing

    partner-repacks