|
|
|
PageOutline
|
|
|
|
|
|
|
|
# Debugging Tor browser on Windows
|
|
|
|
|
|
|
|
The instructions for gcc vs clang couldn't be different, so be sure you know which version you're using.
|
|
|
|
|
|
|
|
# mingw-gcc
|
|
|
|
|
|
|
|
Debugging on Windows is tricky. I'm not aware of a mechanism to get symbols to work. There are several approaches and failed attempts. I'm going to detail them in an Appendix, but I never got any to work. In theory, it should be possible to debug on Windows, with symbols, using gdb. The mingw gcc toolchain can produce debugging information in the DWARF format. When we switch to mingw-clang, it might be possible to generate pdbs in which case we could generate symbols understandable with WinDBG.
|
|
|
|
|
|
|
|
## General
|
|
|
|
|
|
|
|
You'll want to get familiar with the following environment variables for debugging:
|
|
|
|
|
|
|
|
- MOZ_DEBUG_CHILD_PROCESS - should breakpoint child processes, but doesn't really work in Windows
|
|
|
|
- MOZ_DEBUG_CHILD_PAUSE - will pause child processes upon start and print their pid so you can connect to them
|
|
|
|
- MOZ_LOG - for outputting logging information
|
|
|
|
- MOZ_IPC_MESSAGE_LOG - more other logging relating to IPC
|
|
|
|
|
|
|
|
You'll also want to be debugging a build with MOZ_ASSERTs - these are usually what we use to orient ourselves. So make sure you're building with --enable-debug, which is not the default for tor.
|
|
|
|
|
|
|
|
## Debugging with gdb
|
|
|
|
|
|
|
|
The most important thing, if you're going to attempt to debug with gdb is to use a MinGW compiled gdb. You _cannot_ use the cygwin gdb. If you just want a pre-compiled gdb, you can get one here: <http://www.equation.com/servlet/equation.cmd?fa=gdb>
|
|
|
|
|
|
|
|
## Debugging with WinDBG
|
|
|
|
|
|
|
|
WinDBG commands make no sense. Get <http://windbg.info/doc/1-common-cmds.html> open and ready.
|
|
|
|
|
|
|
|
### Working Backwards from a Crash
|
|
|
|
|
|
|
|
First off, we have to figure out if the crash is in the parent or child. Try launching the process through WinDBG and see if you get the crash. Otherwise try attaching to the child (using MOZ_DEBUG_CHILD_PAUSE). Once you have a process broken in WinDBG, **g** will tell it to continue. ('g' for 'go')
|
|
|
|
|
|
|
|
Okay, we've crashed:
|
|
|
|
|
|
|
|
```plaintext
|
|
|
|
*** ERROR: Symbol file could not be found. Defaulted to export symbols for C:\cygwin64\home\Tom\working\ff-transfer\debugging\14 svg logging\xul.dll -
|
|
|
|
xul!XRE_GetBootstrap+0x2508847:
|
|
|
|
00000000`071da737 c7007a000000 mov dword ptr [rax],7Ah ds:00000000`00000000=????????
|
|
|
|
```
|
|
|
|
|
|
|
|
Why did we crash? This code is trying to move 7Ah into the memory address stored in rax. What's in rax? We can look at registers with **r** (for registers.)
|
|
|
|
|
|
|
|
```plaintext
|
|
|
|
0:000> r
|
|
|
|
rax=0000000000000000 rbx=0000000000000001 rcx=00000000ffffffff
|
|
|
|
rdx=0000000000000002 rsi=0000000000000001 rdi=0000000000000000
|
|
|
|
rip=00000000071da737 rsp=00000000006596e0 rbp=0000000000659740
|
|
|
|
r8=00007ffc56c14920 r9=0000000000653280 r10=0000000000000000
|
|
|
|
r11=0000000000000246 r12=0000000000000016 r13=00000000000000b0
|
|
|
|
r14=0000000000000000 r15=0000000000000000
|
|
|
|
iopl=0 nv up ei pl nz na pe nc
|
|
|
|
cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010202
|
|
|
|
xul!XRE_GetBootstrap+0x2508847:
|
|
|
|
00000000`071da737 c7007a000000 mov dword ptr [rax],7Ah ds:00000000`00000000=????????
|
|
|
|
```
|
|
|
|
|
|
|
|
Note that the command I type starts with '0:000>'.
|
|
|
|
|
|
|
|
Okay. rax is 0x0000000000000000 - yea trying to store memory to the nullptr is going to crash. Why is it doing that?!??! (There's an easy answer below, you should learn this signature, but for now let's keep going.)
|
|
|
|
|
|
|
|
Once you've crashed, you can get a stacktrace with **k**.
|
|
|
|
|
|
|
|
Now you'll have a symbol-less stack frame, like:
|
|
|
|
|
|
|
|
```plaintext
|
|
|
|
0:000> k
|
|
|
|
# Child-SP RetAddr Call Site
|
|
|
|
00 00000000`006596e0 00000000`071da80c xul!XRE_GetBootstrap+0x2508847
|
|
|
|
01 00000000`00659750 00000000`05a1be2f xul!XRE_GetBootstrap+0x250891c
|
|
|
|
02 00000000`006597e0 00000000`05a1db47 xul!XRE_GetBootstrap+0xd49f3f
|
|
|
|
03 00000000`00659850 00000000`053a1dc0 xul!XRE_GetBootstrap+0xd4bc57
|
|
|
|
04 00000000`00659880 00000000`0659877e xul!XRE_GetBootstrap+0x6cfed0
|
|
|
|
05 00000000`00659920 00000000`0558e20e xul!XRE_GetBootstrap+0x18c688e
|
|
|
|
06 00000000`006599c0 00000000`0a750e80 xul!XRE_GetBootstrap+0x8bc31e
|
|
|
|
07 00000000`00659cf0 00000000`0678b8bd xul!XRE_GetBootstrap+0x5a7ef90
|
|
|
|
08 00000000`00659d20 00000000`0653ee6b xul!XRE_GetBootstrap+0x1ab99cd
|
|
|
|
09 00000000`00659d50 00000000`059eedb4 xul!XRE_GetBootstrap+0x186cf7b
|
|
|
|
0a 00000000`00659d80 00000000`07cf7eac xul!XRE_GetBootstrap+0xd1cec4
|
|
|
|
0b 00000000`00659dd0 00000000`07cf59a8 xul!XRE_GetBootstrap+0x3025fbc
|
|
|
|
0c 00000000`00659f20 00000000`059ece67 xul!XRE_GetBootstrap+0x3023ab8
|
|
|
|
0d 00000000`00659fc0 00000000`059ee15d xul!XRE_GetBootstrap+0xd1af77
|
|
|
|
0e 00000000`0065a490 00000000`05a18cd1 xul!XRE_GetBootstrap+0xd1c26d
|
|
|
|
0f 00000000`0065a690 00000000`0a173d1c xul!XRE_GetBootstrap+0xd46de1
|
|
|
|
10 00000000`0065d9b0 00000000`05c8d432 xul!XRE_GetBootstrap+0x54a1e2c
|
|
|
|
11 00000000`0065dc70 00000000`05c8d998 xul!XRE_GetBootstrap+0xfbb542
|
|
|
|
12 00000000`0065dd00 00000000`05c8cecf xul!XRE_GetBootstrap+0xfbbaa8
|
|
|
|
13 00000000`0065ddc0 00000000`05ed18ea xul!XRE_GetBootstrap+0xfbafdf
|
|
|
|
14 00000000`0065de00 00000000`07da1f11 xul!XRE_GetBootstrap+0x11ff9fa
|
|
|
|
15 00000000`0065e3d0 00000000`07da243a xul!XRE_GetBootstrap+0x30d0021
|
|
|
|
16 00000000`0065e450 00000000`07da2ae3 xul!XRE_GetBootstrap+0x30d054a
|
|
|
|
17 00000000`0065e4f0 00000000`07ebf569 xul!XRE_GetBootstrap+0x30d0bf3
|
|
|
|
18 00000000`0065e580 00000000`07ebfaf2 xul!XRE_GetBootstrap+0x31ed679
|
|
|
|
19 00000000`0065e630 00000000`07ebf8bf xul!XRE_GetBootstrap+0x31edc02
|
|
|
|
1a 00000000`0065e6e0 00000000`09f406f8 xul!XRE_GetBootstrap+0x31ed9cf
|
|
|
|
1b 00000000`0065e7b0 00000000`09f41465 xul!XRE_GetBootstrap+0x526e808
|
|
|
|
1c 00000000`0065e800 00000000`0950b127 xul!XRE_GetBootstrap+0x526f575
|
|
|
|
1d 00000000`0065e8f0 00000000`09447b25 xul!XRE_GetBootstrap+0x4839237
|
|
|
|
1e 00000000`0065ec20 00000000`09445996 xul!XRE_GetBootstrap+0x4775c35
|
|
|
|
1f 00000000`0065ec80 00000000`0944352e xul!XRE_GetBootstrap+0x4773aa6
|
|
|
|
20 00000000`0065ee00 00000000`0944381e xul!XRE_GetBootstrap+0x477163e
|
|
|
|
21 00000000`0065ee50 00000000`0a4915eb xul!XRE_GetBootstrap+0x477192e
|
|
|
|
22 00000000`0065eea0 00000000`04f08cd1 xul!XRE_GetBootstrap+0x57bf6fb
|
|
|
|
23 00000000`0065f560 00000000`09435c16 xul!XRE_GetBootstrap+0x236de1
|
|
|
|
24 00000000`0065f5a0 00000000`09542e02 xul!XRE_GetBootstrap+0x4763d26
|
|
|
|
25 00000000`0065f5f0 00000000`055ace38 xul!XRE_GetBootstrap+0x4870f12
|
|
|
|
26 00000000`0065f640 00000000`055ac6d8 xul!XRE_GetBootstrap+0x8daf48
|
|
|
|
27 00000000`0065f690 00000000`055adc35 xul!XRE_GetBootstrap+0x8da7e8
|
|
|
|
28 00000000`0065f6d0 00000000`05d1ec56 xul!XRE_GetBootstrap+0x8dbd45
|
|
|
|
29 00000000`0065f720 00000000`054cff1b xul!XRE_GetBootstrap+0x104cd66
|
|
|
|
2a 00000000`0065f770 00000000`04dbe9cc xul!XRE_GetBootstrap+0x7fe02b
|
|
|
|
2b 00000000`0065f7b0 00000000`09542cb4 xul!XRE_GetBootstrap+0xecadc
|
|
|
|
2c 00000000`0065f810 00000000`055ace38 xul!XRE_GetBootstrap+0x4870dc4
|
|
|
|
2d 00000000`0065f860 00000000`055ac6d8 xul!XRE_GetBootstrap+0x8daf48
|
|
|
|
2e 00000000`0065f8b0 00000000`055adc35 xul!XRE_GetBootstrap+0x8da7e8
|
|
|
|
2f 00000000`0065f8f0 00000000`04f28756 xul!XRE_GetBootstrap+0x8dbd45
|
|
|
|
30 00000000`0065f940 00000000`07afa1bc xul!XRE_GetBootstrap+0x256866
|
|
|
|
*** WARNING: Unable to verify timestamp for C:\cygwin64\home\Tom\working\ff-transfer\debugging\14 svg logging\firefox.exe
|
|
|
|
*** ERROR: Module load completed but symbols could not be loaded for C:\cygwin64\home\Tom\working\ff-transfer\debugging\14 svg logging\firefox.exe
|
|
|
|
31 00000000`0065fca0 00000000`0040a745 xul!XRE_GetBootstrap+0x2e282cc
|
|
|
|
32 00000000`0065fcd0 00000000`0040a5bc firefox+0xa745
|
|
|
|
33 00000000`0065fd10 00000000`0040cffe firefox+0xa5bc
|
|
|
|
34 00000000`0065fd90 00000000`0040cdfc firefox+0xcffe
|
|
|
|
35 00000000`0065fe10 00000000`004013c8 firefox+0xcdfc
|
|
|
|
36 00000000`0065fe60 00000000`004014eb firefox+0x13c8
|
|
|
|
37 00000000`0065ff30 00007ffc`55f31fe4 firefox+0x14eb
|
|
|
|
38 00000000`0065ff60 00007ffc`5839efc1 KERNEL32BaseThreadInitThunk+0x14
|
|
|
|
39 00000000`0065ff90 00000000`00000000 ntdllRtlUserThreadStart+0x21
|
|
|
|
```
|
|
|
|
|
|
|
|
Okay, time to explore the frames. Our primary goal is to understand where we are. To do that, we're going to go up the frame until we find something with an ASSERT or a LOG. Basically, we're looking for something that's referencing a hardcoded string we can search for. ASSERTs tend to be more common.
|
|
|
|
|
|
|
|
First, let's look at our crashing frame. That's 'xul!XRE_GetBootstrap+0x2508847'. We want to disassemble the code we are crashing at. We can do what with **u** (u for unassemble.) This will show the assembly code that we crash on:
|
|
|
|
|
|
|
|
```plaintext
|
|
|
|
0:000> u xul!XRE_GetBootstrap+0x2508847
|
|
|
|
xul!XRE_GetBootstrap+0x2508847:
|
|
|
|
00000000`071da737 c7007a000000 mov dword ptr [rax],7Ah
|
|
|
|
00000000`071da73d e846e69ffd call xul+0x4e8d88 (00000000`04bd8d88)
|
|
|
|
00000000`071da742 488b4510 mov rax,qword ptr [rbp+10h]
|
|
|
|
00000000`071da746 488b5518 mov rdx,qword ptr [rbp+18h]
|
|
|
|
00000000`071da74a 4889c1 mov rcx,rax
|
|
|
|
00000000`071da74d e82e89ce03 call xul!XRE_GetBootstrap+0x61f1190 (00000000`0aec3080)
|
|
|
|
00000000`071da752 488b4510 mov rax,qword ptr [rbp+10h]
|
|
|
|
00000000`071da756 4883c460 add rsp,60h
|
|
|
|
```
|
|
|
|
|
|
|
|
Notice that although I ask it to unassemble a relative address, the memory address it shows me is absolute: 071da737.
|
|
|
|
|
|
|
|
This shows us the crashing instruction and the next few we would execute, but really we'd like to get more context. So we're going to unassemble the entire function of this frame with **uf**.
|
|
|
|
|
|
|
|
```plaintext
|
|
|
|
0:000> uf xul!XRE_GetBootstrap+0x2508847
|
|
|
|
xul!XRE_GetBootstrap+0x2508680:
|
|
|
|
00000000`071da570 55 push rbp
|
|
|
|
00000000`071da571 4889e5 mov rbp,rsp
|
|
|
|
00000000`071da574 4883ec60 sub rsp,60h
|
|
|
|
00000000`071da578 48894d10 mov qword ptr [rbp+10h],rcx
|
|
|
|
00000000`071da57c 48895518 mov qword ptr [rbp+18h],rdx
|
|
|
|
00000000`071da580 4489c0 mov eax,r8d
|
|
|
|
00000000`071da583 884520 mov byte ptr [rbp+20h],al
|
|
|
|
00000000`071da586 e8b5bc2703 call xul!XRE_GetBootstrap+0x5784350 (00000000`0a456240)
|
|
|
|
00000000`071da58b 4889c2 mov rdx,rax
|
|
|
|
00000000`071da58e 488b4518 mov rax,qword ptr [rbp+18h]
|
|
|
|
00000000`071da592 41b800000000 mov r8d,0
|
|
|
|
00000000`071da598 4889c1 mov rcx,rax
|
|
|
|
00000000`071da59b e8d036ce03 call xul!XRE_GetBootstrap+0x61ebd80 (00000000`0aebdc70)
|
|
|
|
00000000`071da5a0 488945f8 mov qword ptr [rbp-8],rax
|
|
|
|
00000000`071da5a4 48837df800 cmp qword ptr [rbp-8],0
|
|
|
|
00000000`071da5a9 7461 je xul!XRE_GetBootstrap+0x250871c (00000000`071da60c) Branch
|
|
|
|
|
|
|
|
xul!XRE_GetBootstrap+0x25086bb:
|
|
|
|
00000000`071da5ab 488d0d9eb7fb04 lea rcx,[xul!workerlz4_maxCompressedSize+0x31a70 (00000000`0c195d50)]
|
|
|
|
00000000`071da5b2 e859eb9600 call xul!XRE_GetBootstrap+0x2e77220 (00000000`07b49110)
|
|
|
|
00000000`071da5b7 488945f0 mov qword ptr [rbp-10h],rax
|
|
|
|
00000000`071da5bb 488b45f0 mov rax,qword ptr [rbp-10h]
|
|
|
|
00000000`071da5bf ba04000000 mov edx,4
|
|
|
|
00000000`071da5c4 4889c1 mov rcx,rax
|
|
|
|
00000000`071da5c7 e804c6b002 call xul!XRE_GetBootstrap+0x5014ce0 (00000000`09ce6bd0)
|
|
|
|
00000000`071da5cc 84c0 test al,al
|
|
|
|
00000000`071da5ce 7427 je xul!XRE_GetBootstrap+0x2508707 (00000000`071da5f7) Branch
|
|
|
|
|
|
|
|
xul!XRE_GetBootstrap+0x25086e0:
|
|
|
|
00000000`071da5d0 0fb65520 movzx edx,byte ptr [rbp+20h]
|
|
|
|
00000000`071da5d4 488b4d18 mov rcx,qword ptr [rbp+18h]
|
|
|
|
00000000`071da5d8 488b45f0 mov rax,qword ptr [rbp-10h]
|
|
|
|
00000000`071da5dc 89542420 mov dword ptr [rsp+20h],edx
|
|
|
|
00000000`071da5e0 4989c9 mov r9,rcx
|
|
|
|
00000000`071da5e3 4c8d058e078905 lea r8,[xul!workerlz4_maxCompressedSize+0x906a98 (00000000`0ca6ad78)]
|
|
|
|
00000000`071da5ea ba04000000 mov edx,4
|
|
|
|
00000000`071da5ef 4889c1 mov rcx,rax
|
|
|
|
00000000`071da5f2 e81932b302 call xul!XRE_GetBootstrap+0x503b920 (00000000`09d0d810)
|
|
|
|
|
|
|
|
xul!XRE_GetBootstrap+0x2508707:
|
|
|
|
00000000`071da5f7 488b45f8 mov rax,qword ptr [rbp-8]
|
|
|
|
00000000`071da5fb 4889c2 mov rdx,rax
|
|
|
|
00000000`071da5fe 488b4d10 mov rcx,qword ptr [rbp+10h]
|
|
|
|
00000000`071da602 e829b44e00 call xul!XRE_GetBootstrap+0x29f3b40 (00000000`076c5a30)
|
|
|
|
00000000`071da607 e946010000 jmp xul!XRE_GetBootstrap+0x2508862 (00000000`071da752) Branch
|
|
|
|
|
|
|
|
xul!XRE_GetBootstrap+0x250871c:
|
|
|
|
00000000`071da60c 488d0d3db7fb04 lea rcx,[xul!workerlz4_maxCompressedSize+0x31a70 (00000000`0c195d50)]
|
|
|
|
00000000`071da613 e8f8ea9600 call xul!XRE_GetBootstrap+0x2e77220 (00000000`07b49110)
|
|
|
|
00000000`071da618 488945e8 mov qword ptr [rbp-18h],rax
|
|
|
|
00000000`071da61c 488b45e8 mov rax,qword ptr [rbp-18h]
|
|
|
|
00000000`071da620 ba04000000 mov edx,4
|
|
|
|
00000000`071da625 4889c1 mov rcx,rax
|
|
|
|
00000000`071da628 e8a3c5b002 call xul!XRE_GetBootstrap+0x5014ce0 (00000000`09ce6bd0)
|
|
|
|
00000000`071da62d 84c0 test al,al
|
|
|
|
00000000`071da62f 7427 je xul!XRE_GetBootstrap+0x2508768 (00000000`071da658) Branch
|
|
|
|
|
|
|
|
xul!XRE_GetBootstrap+0x2508741:
|
|
|
|
00000000`071da631 0fb65520 movzx edx,byte ptr [rbp+20h]
|
|
|
|
00000000`071da635 488b4d18 mov rcx,qword ptr [rbp+18h]
|
|
|
|
00000000`071da639 488b45e8 mov rax,qword ptr [rbp-18h]
|
|
|
|
00000000`071da63d 89542420 mov dword ptr [rsp+20h],edx
|
|
|
|
00000000`071da641 4989c9 mov r9,rcx
|
|
|
|
00000000`071da644 4c8d05ad078905 lea r8,[xul!workerlz4_maxCompressedSize+0x906b18 (00000000`0ca6adf8)]
|
|
|
|
00000000`071da64b ba04000000 mov edx,4
|
|
|
|
00000000`071da650 4889c1 mov rcx,rax
|
|
|
|
00000000`071da653 e8b831b302 call xul!XRE_GetBootstrap+0x503b920 (00000000`09d0d810)
|
|
|
|
|
|
|
|
xul!XRE_GetBootstrap+0x2508768:
|
|
|
|
00000000`071da658 488b4518 mov rax,qword ptr [rbp+18h]
|
|
|
|
00000000`071da65c 4889c1 mov rcx,rax
|
|
|
|
00000000`071da65f e8fcd73bff call xul!XRE_GetBootstrap+0x18c5f70 (00000000`06597e60)
|
|
|
|
00000000`071da664 84c0 test al,al
|
|
|
|
00000000`071da666 7412 je xul!XRE_GetBootstrap+0x250878a (00000000`071da67a) Branch
|
|
|
|
|
|
|
|
xul!XRE_GetBootstrap+0x2508778:
|
|
|
|
00000000`071da668 0fb64520 movzx eax,byte ptr [rbp+20h]
|
|
|
|
00000000`071da66c 83f001 xor eax,1
|
|
|
|
00000000`071da66f 84c0 test al,al
|
|
|
|
00000000`071da671 7407 je xul!XRE_GetBootstrap+0x250878a (00000000`071da67a) Branch
|
|
|
|
|
|
|
|
xul!XRE_GetBootstrap+0x2508783:
|
|
|
|
00000000`071da673 b801000000 mov eax,1
|
|
|
|
00000000`071da678 eb05 jmp xul!XRE_GetBootstrap+0x250878f (00000000`071da67f) Branch
|
|
|
|
|
|
|
|
xul!XRE_GetBootstrap+0x250878a:
|
|
|
|
00000000`071da67a b800000000 mov eax,0
|
|
|
|
|
|
|
|
xul!XRE_GetBootstrap+0x250878f:
|
|
|
|
00000000`071da67f 84c0 test al,al
|
|
|
|
00000000`071da681 0f84bb000000 je xul!XRE_GetBootstrap+0x2508852 (00000000`071da742) Branch
|
|
|
|
|
|
|
|
xul!XRE_GetBootstrap+0x2508797:
|
|
|
|
00000000`071da687 e8140d2803 call xul!XRE_GetBootstrap+0x57894b0 (00000000`0a45b3a0)
|
|
|
|
00000000`071da68c 4889c2 mov rdx,rax
|
|
|
|
00000000`071da68f 488b4518 mov rax,qword ptr [rbp+18h]
|
|
|
|
00000000`071da693 41b800000000 mov r8d,0
|
|
|
|
00000000`071da699 4889c1 mov rcx,rax
|
|
|
|
00000000`071da69c e88f31ce03 call xul!XRE_GetBootstrap+0x61eb940 (00000000`0aebd830)
|
|
|
|
00000000`071da6a1 488945e0 mov qword ptr [rbp-20h],rax
|
|
|
|
00000000`071da6a5 488d0da4b6fb04 lea rcx,[xul!workerlz4_maxCompressedSize+0x31a70 (00000000`0c195d50)]
|
|
|
|
00000000`071da6ac e85fea9600 call xul!XRE_GetBootstrap+0x2e77220 (00000000`07b49110)
|
|
|
|
00000000`071da6b1 488945d8 mov qword ptr [rbp-28h],rax
|
|
|
|
00000000`071da6b5 488b45d8 mov rax,qword ptr [rbp-28h]
|
|
|
|
00000000`071da6b9 ba04000000 mov edx,4
|
|
|
|
00000000`071da6be 4889c1 mov rcx,rax
|
|
|
|
00000000`071da6c1 e80ac5b002 call xul!XRE_GetBootstrap+0x5014ce0 (00000000`09ce6bd0)
|
|
|
|
00000000`071da6c6 84c0 test al,al
|
|
|
|
00000000`071da6c8 7440 je xul!XRE_GetBootstrap+0x250881a (00000000`071da70a) Branch
|
|
|
|
|
|
|
|
xul!XRE_GetBootstrap+0x25087da:
|
|
|
|
00000000`071da6ca 48837de000 cmp qword ptr [rbp-20h],0
|
|
|
|
00000000`071da6cf 7509 jne xul!XRE_GetBootstrap+0x25087ea (00000000`071da6da) Branch
|
|
|
|
|
|
|
|
xul!XRE_GetBootstrap+0x25087e1:
|
|
|
|
00000000`071da6d1 488d0580078905 lea rax,[xul!workerlz4_maxCompressedSize+0x906b78 (00000000`0ca6ae58)]
|
|
|
|
00000000`071da6d8 eb07 jmp xul!XRE_GetBootstrap+0x25087f1 (00000000`071da6e1) Branch
|
|
|
|
|
|
|
|
xul!XRE_GetBootstrap+0x25087ea:
|
|
|
|
00000000`071da6da 488d057f078905 lea rax,[xul!workerlz4_maxCompressedSize+0x906b80 (00000000`0ca6ae60)]
|
|
|
|
|
|
|
|
xul!XRE_GetBootstrap+0x25087f1:
|
|
|
|
00000000`071da6e1 0fb65520 movzx edx,byte ptr [rbp+20h]
|
|
|
|
00000000`071da6e5 4c8b4518 mov r8,qword ptr [rbp+18h]
|
|
|
|
00000000`071da6e9 488b4dd8 mov rcx,qword ptr [rbp-28h]
|
|
|
|
00000000`071da6ed 4889442428 mov qword ptr [rsp+28h],rax
|
|
|
|
00000000`071da6f2 89542420 mov dword ptr [rsp+20h],edx
|
|
|
|
00000000`071da6f6 4d89c1 mov r9,r8
|
|
|
|
00000000`071da6f9 4c8d0568078905 lea r8,[xul!workerlz4_maxCompressedSize+0x906b88 (00000000`0ca6ae68)]
|
|
|
|
00000000`071da700 ba04000000 mov edx,4
|
|
|
|
00000000`071da705 e80631b302 call xul!XRE_GetBootstrap+0x503b920 (00000000`09d0d810)
|
|
|
|
|
|
|
|
xul!XRE_GetBootstrap+0x250881a:
|
|
|
|
00000000`071da70a 48837de000 cmp qword ptr [rbp-20h],0
|
|
|
|
00000000`071da70f 0f95c0 setne al
|
|
|
|
00000000`071da712 0fb6c0 movzx eax,al
|
|
|
|
00000000`071da715 85c0 test eax,eax
|
|
|
|
00000000`071da717 7429 je xul!XRE_GetBootstrap+0x2508852 (00000000`071da742) Branch
|
|
|
|
|
|
|
|
xul!XRE_GetBootstrap+0x2508829:
|
|
|
|
00000000`071da719 41b87a000000 mov r8d,7Ah
|
|
|
|
00000000`071da71f 488d156a058905 lea rdx,[xul!workerlz4_maxCompressedSize+0x9069b0 (00000000`0ca6ac90)]
|
|
|
|
00000000`071da726 488d0dbb078905 lea rcx,[xul!workerlz4_maxCompressedSize+0x906c08 (00000000`0ca6aee8)]
|
|
|
|
00000000`071da72d e8eeb3a7fd call xul!DumpJSStack+0x58fc0 (00000000`04c55b20)
|
|
|
|
00000000`071da732 b800000000 mov eax,0
|
|
|
|
00000000`071da737 c7007a000000 mov dword ptr [rax],7Ah
|
|
|
|
00000000`071da73d e846e69ffd call xul+0x4e8d88 (00000000`04bd8d88)
|
|
|
|
|
|
|
|
xul!XRE_GetBootstrap+0x2508852:
|
|
|
|
00000000`071da742 488b4510 mov rax,qword ptr [rbp+10h]
|
|
|
|
00000000`071da746 488b5518 mov rdx,qword ptr [rbp+18h]
|
|
|
|
00000000`071da74a 4889c1 mov rcx,rax
|
|
|
|
00000000`071da74d e82e89ce03 call xul!XRE_GetBootstrap+0x61f1190 (00000000`0aec3080)
|
|
|
|
|
|
|
|
xul!XRE_GetBootstrap+0x2508862:
|
|
|
|
00000000`071da752 488b4510 mov rax,qword ptr [rbp+10h]
|
|
|
|
00000000`071da756 4883c460 add rsp,60h
|
|
|
|
00000000`071da75a 5d pop rbp
|
|
|
|
00000000`071da75b c3 ret
|
|
|
|
```
|
|
|
|
|
|
|
|
Note that the first line after the 'uf' instruction shows us the _start_ of the function. xul!XRE_GetBootstrap+0x2508847 is where we crashed; but xul!XRE_GetBootstrap+0x2508680 is where the function began. And the absolute address of xul!XRE_GetBootstrap+0x2508847 is 071da737 - if you control+f for 071da737 on this page you'll see the same instructions we unassembled.
|
|
|
|
|
|
|
|
Also: why does it break the function up with newlines in between? Each of those sections of the function is part of a branch. In IDA you'd get a nice graph view like so: <http://hexblog.com/ida_pro/pix/idaqt_preview_100310_1.html> (picture is an illustrative example, not this particular function). Take 'xul!XRE_GetBootstrap+0x2508852' for example. If you control+f on this page for 0x2508852, you'll see there are a couple of places in this function we jump to this location.
|
|
|
|
|
|
|
|
Now, let's pretend this frame doesn't tell us anything useful. We'd need to look at the address of one frame up: xul!XRE_GetBootstrap+0x250891c
|
|
|
|
|
|
|
|
```plaintext
|
|
|
|
0:000> u xul!XRE_GetBootstrap+0x250891c
|
|
|
|
xul!XRE_GetBootstrap+0x250891c:
|
|
|
|
00000000`071da80c 488b4510 mov rax,qword ptr [rbp+10h]
|
|
|
|
00000000`071da810 488b4810 mov rcx,qword ptr [rax+10h]
|
|
|
|
00000000`071da814 488d45d8 lea rax,[rbp-28h]
|
|
|
|
00000000`071da818 488b5518 mov rdx,qword ptr [rbp+18h]
|
|
|
|
00000000`071da81c 4989c8 mov r8,rcx
|
|
|
|
00000000`071da81f 4889c1 mov rcx,rax
|
|
|
|
00000000`071da822 e8892cce03 call xul!XRE_GetBootstrap+0x61eb5c0 (00000000`0aebd4b0)
|
|
|
|
00000000`071da827 488d55b0 lea rdx,[rbp-50h]
|
|
|
|
```
|
|
|
|
|
|
|
|
In this situation we see the instruction immediately following the 'call' we made to go one stack frame down. Again, we need more context, so disassemble the whole function with **uf**.
|
|
|
|
|
|
|
|
```plaintext
|
|
|
|
0:000> uf xul!XRE_GetBootstrap+0x250891c
|
|
|
|
xul!XRE_GetBootstrap+0x2508870:
|
|
|
|
00000000`071da760 55 push rbp
|
|
|
|
00000000`071da761 4889e5 mov rbp,rsp
|
|
|
|
00000000`071da764 4883c480 add rsp,0FFFFFFFFFFFFFF80h
|
|
|
|
00000000`071da768 48894d10 mov qword ptr [rbp+10h],rcx
|
|
|
|
00000000`071da76c 48895518 mov qword ptr [rbp+18h],rdx
|
|
|
|
00000000`071da770 488d0dd9b5fb04 lea rcx,[xul!workerlz4_maxCompressedSize+0x31a70 (00000000`0c195d50)]
|
|
|
|
00000000`071da777 e894e99600 call xul!XRE_GetBootstrap+0x2e77220 (00000000`07b49110)
|
|
|
|
00000000`071da77c 488945f8 mov qword ptr [rbp-8],rax
|
|
|
|
00000000`071da780 488b45f8 mov rax,qword ptr [rbp-8]
|
|
|
|
00000000`071da784 ba04000000 mov edx,4
|
|
|
|
00000000`071da789 4889c1 mov rcx,rax
|
|
|
|
00000000`071da78c e83fc4b002 call xul!XRE_GetBootstrap+0x5014ce0 (00000000`09ce6bd0)
|
|
|
|
00000000`071da791 84c0 test al,al
|
|
|
|
00000000`071da793 7435 je xul!XRE_GetBootstrap+0x25088da (00000000`071da7ca) Branch
|
|
|
|
|
|
|
|
xul!XRE_GetBootstrap+0x25088a5:
|
|
|
|
00000000`071da795 488b4510 mov rax,qword ptr [rbp+10h]
|
|
|
|
00000000`071da799 488b4018 mov rax,qword ptr [rax+18h]
|
|
|
|
00000000`071da79d 483b4518 cmp rax,qword ptr [rbp+18h]
|
|
|
|
00000000`071da7a1 0f94c0 sete al
|
|
|
|
00000000`071da7a4 0fb6d0 movzx edx,al
|
|
|
|
00000000`071da7a7 488b4d18 mov rcx,qword ptr [rbp+18h]
|
|
|
|
00000000`071da7ab 488b45f8 mov rax,qword ptr [rbp-8]
|
|
|
|
00000000`071da7af 89542420 mov dword ptr [rsp+20h],edx
|
|
|
|
00000000`071da7b3 4989c9 mov r9,rcx
|
|
|
|
00000000`071da7b6 4c8d0573058905 lea r8,[xul!workerlz4_maxCompressedSize+0x906a50 (00000000`0ca6ad30)]
|
|
|
|
00000000`071da7bd ba04000000 mov edx,4
|
|
|
|
00000000`071da7c2 4889c1 mov rcx,rax
|
|
|
|
00000000`071da7c5 e84630b302 call xul!XRE_GetBootstrap+0x503b920 (00000000`09d0d810)
|
|
|
|
|
|
|
|
xul!XRE_GetBootstrap+0x25088da:
|
|
|
|
00000000`071da7ca 488b4510 mov rax,qword ptr [rbp+10h]
|
|
|
|
00000000`071da7ce 488b4018 mov rax,qword ptr [rax+18h]
|
|
|
|
00000000`071da7d2 483b4518 cmp rax,qword ptr [rbp+18h]
|
|
|
|
00000000`071da7d6 7516 jne xul!XRE_GetBootstrap+0x25088fe (00000000`071da7ee) Branch
|
|
|
|
|
|
|
|
xul!XRE_GetBootstrap+0x25088e8:
|
|
|
|
00000000`071da7d8 488b4510 mov rax,qword ptr [rbp+10h]
|
|
|
|
00000000`071da7dc 488b5020 mov rdx,qword ptr [rax+20h]
|
|
|
|
00000000`071da7e0 488d45b0 lea rax,[rbp-50h]
|
|
|
|
00000000`071da7e4 4889c1 mov rcx,rax
|
|
|
|
00000000`071da7e7 e844b24e00 call xul!XRE_GetBootstrap+0x29f3b40 (00000000`076c5a30)
|
|
|
|
00000000`071da7ec eb1e jmp xul!XRE_GetBootstrap+0x250891c (00000000`071da80c) Branch
|
|
|
|
|
|
|
|
xul!XRE_GetBootstrap+0x25088fe:
|
|
|
|
00000000`071da7ee 488b4510 mov rax,qword ptr [rbp+10h]
|
|
|
|
00000000`071da7f2 0fb64038 movzx eax,byte ptr [rax+38h]
|
|
|
|
00000000`071da7f6 0fb6c8 movzx ecx,al
|
|
|
|
00000000`071da7f9 488d45b0 lea rax,[rbp-50h]
|
|
|
|
00000000`071da7fd 488b5518 mov rdx,qword ptr [rbp+18h]
|
|
|
|
00000000`071da801 4189c8 mov r8d,ecx
|
|
|
|
00000000`071da804 4889c1 mov rcx,rax
|
|
|
|
00000000`071da807 e864fdffff call xul!XRE_GetBootstrap+0x2508680 (00000000`071da570)
|
|
|
|
|
|
|
|
xul!XRE_GetBootstrap+0x250891c:
|
|
|
|
00000000`071da80c 488b4510 mov rax,qword ptr [rbp+10h]
|
|
|
|
00000000`071da810 488b4810 mov rcx,qword ptr [rax+10h]
|
|
|
|
00000000`071da814 488d45d8 lea rax,[rbp-28h]
|
|
|
|
00000000`071da818 488b5518 mov rdx,qword ptr [rbp+18h]
|
|
|
|
00000000`071da81c 4989c8 mov r8,rcx
|
|
|
|
00000000`071da81f 4889c1 mov rcx,rax
|
|
|
|
00000000`071da822 e8892cce03 call xul!XRE_GetBootstrap+0x61eb5c0 (00000000`0aebd4b0)
|
|
|
|
00000000`071da827 488d55b0 lea rdx,[rbp-50h]
|
|
|
|
00000000`071da82b 488d45e0 lea rax,[rbp-20h]
|
|
|
|
00000000`071da82f 4889c1 mov rcx,rax
|
|
|
|
00000000`071da832 e8f9b14e00 call xul!XRE_GetBootstrap+0x29f3b40 (00000000`076c5a30)
|
|
|
|
00000000`071da837 488d45c0 lea rax,[rbp-40h]
|
|
|
|
00000000`071da83b 488d4dd8 lea rcx,[rbp-28h]
|
|
|
|
00000000`071da83f 488d55e0 lea rdx,[rbp-20h]
|
|
|
|
00000000`071da843 4989c8 mov r8,rcx
|
|
|
|
00000000`071da846 4889c1 mov rcx,rax
|
|
|
|
00000000`071da849 e8d24f2002 call xul!XRE_GetBootstrap+0x470d930 (00000000`093df820)
|
|
|
|
00000000`071da84e 488b4510 mov rax,qword ptr [rbp+10h]
|
|
|
|
00000000`071da852 488d5028 lea rdx,[rax+28h]
|
|
|
|
00000000`071da856 488b4510 mov rax,qword ptr [rbp+10h]
|
|
|
|
00000000`071da85a 4883c028 add rax,28h
|
|
|
|
00000000`071da85e 488d4dc0 lea rcx,[rbp-40h]
|
|
|
|
00000000`071da862 4989c8 mov r8,rcx
|
|
|
|
00000000`071da865 4889c1 mov rcx,rax
|
|
|
|
00000000`071da868 e813b14e00 call xul!XRE_GetBootstrap+0x29f3a90 (00000000`076c5980)
|
|
|
|
00000000`071da86d 488d45c0 lea rax,[rbp-40h]
|
|
|
|
00000000`071da871 4889c1 mov rcx,rax
|
|
|
|
00000000`071da874 e8a7b24e00 call xul!XRE_GetBootstrap+0x29f3c30 (00000000`076c5b20)
|
|
|
|
00000000`071da879 488d45e0 lea rax,[rbp-20h]
|
|
|
|
00000000`071da87d 4889c1 mov rcx,rax
|
|
|
|
00000000`071da880 e89bb24e00 call xul!XRE_GetBootstrap+0x29f3c30 (00000000`076c5b20)
|
|
|
|
00000000`071da885 488d45b0 lea rax,[rbp-50h]
|
|
|
|
00000000`071da889 4889c1 mov rcx,rax
|
|
|
|
00000000`071da88c e88fb24e00 call xul!XRE_GetBootstrap+0x29f3c30 (00000000`076c5b20)
|
|
|
|
00000000`071da891 90 nop
|
|
|
|
00000000`071da892 4883ec80 sub rsp,0FFFFFFFFFFFFFF80h
|
|
|
|
00000000`071da896 5d pop rbp
|
|
|
|
00000000`071da897 c3 ret
|
|
|
|
```
|
|
|
|
|
|
|
|
Now, what we're looking for is a sequence of instructions like the below. We actually hit this in the first stack frame, I just wanted to demonstrate going up a stack frame.
|
|
|
|
|
|
|
|
```plaintext
|
|
|
|
xul!XRE_GetBootstrap+0x2508829: <----- START OF A BRANCH, USUALLY PRESENT
|
|
|
|
00000000`071da719 41b87a000000 mov r8d,7Ah <------ MOV OF A CONSTANT INTO A REGISTER
|
|
|
|
00000000`071da71f 488d156a058905 lea rdx,[xul!workerlz4_maxCompressedSize+0x9069b0 (00000000`0ca6ac90)] <-------- LEA OF A MEMORY ADDRESS INTO A REGISTER
|
|
|
|
00000000`071da726 488d0dbb078905 lea rcx,[xul!workerlz4_maxCompressedSize+0x906c08 (00000000`0ca6aee8)] <-------- LEA OF A MEMORY ADDRESS INTO A REGISTER
|
|
|
|
00000000`071da72d e8eeb3a7fd call xul!DumpJSStack+0x58fc0 (00000000`04c55b20) <-------- CALL
|
|
|
|
00000000`071da732 b800000000 mov eax,0 <-------- MOV 0 INTO eax OR rax
|
|
|
|
00000000`071da737 c7007a000000 mov dword ptr [rax],7Ah <-------- MOV THE CONSTANT INTO THE MEMORY ADDRESS OF THAT REGISTER (WHICH IS A nullptr)
|
|
|
|
```
|
|
|
|
|
|
|
|
This pattern is a MOZ_ASSERT. Here's what it means:
|
|
|
|
|
|
|
|
```plaintext
|
|
|
|
00000000`071da719 41b87a000000 mov r8d,7Ah <------ THIS CONSTANT IS THE LINE NUMBER (in hex)
|
|
|
|
00000000`071da71f 488d156a058905 lea rdx,[xul!workerlz4_maxCompressedSize+0x9069b0 (00000000`0ca6ac90)] <-------- THIS STRING IS THE FILE
|
|
|
|
00000000`071da726 488d0dbb078905 lea rcx,[xul!workerlz4_maxCompressedSize+0x906c08 (00000000`0ca6aee8)] <-------- THIS STRING IS THE ASSERT LINE
|
|
|
|
...
|
|
|
|
...
|
|
|
|
00000000`071da737 c7007a000000 mov dword ptr [rax],7Ah <-------- INTENTIONALLY CRASH THE PROCESS
|
|
|
|
```
|
|
|
|
|
|
|
|
Those lea instructions are loading a hardcoded string. Let's examine that. To do so we're going to use the **d** command for reading memory.
|
|
|
|
|
|
|
|
```plaintext
|
|
|
|
db <addr> - shows raw memory
|
|
|
|
|
|
|
|
0:000> db xul!workerlz4_maxCompressedSize+0x9069b0
|
|
|
|
00000000`0ca6ac90 2f 62 75 69 6c 64 73 2f-77 6f 72 6b 65 72 2f 77 /builds/worker/w
|
|
|
|
00000000`0ca6aca0 6f 72 6b 73 70 61 63 65-2f 62 75 69 6c 64 2f 73 orkspace/build/s
|
|
|
|
00000000`0ca6acb0 72 63 2f 6c 61 79 6f 75-74 2f 73 76 67 2f 6e 73 rc/layout/svg/ns
|
|
|
|
00000000`0ca6acc0 53 56 47 49 6e 74 65 67-72 61 74 69 6f 6e 55 74 SVGIntegrationUt
|
|
|
|
00000000`0ca6acd0 69 6c 73 2e 63 70 70 00-21 6d 46 69 72 73 74 43 ils.cpp.!mFirstC
|
|
|
|
00000000`0ca6ace0 6f 6e 74 69 6e 75 61 74-69 6f 6e 2d 3e 47 65 74 ontinuation->Get
|
|
|
|
00000000`0ca6acf0 50 72 65 76 43 6f 6e 74-69 6e 75 61 74 69 6f 6e PrevContinuation
|
|
|
|
00000000`0ca6ad00 28 29 00 00 00 00 00 00-57 65 20 77 61 6e 74 20 ()......We want
|
|
|
|
|
|
|
|
|
|
|
|
da <addr> - interpret data as an ASCII string
|
|
|
|
|
|
|
|
0:000> da xul!workerlz4_maxCompressedSize+0x9069b0
|
|
|
|
00000000`0ca6ac90 "/builds/worker/workspace/build/s"
|
|
|
|
00000000`0ca6acb0 "rc/layout/svg/nsSVGIntegrationUt"
|
|
|
|
00000000`0ca6acd0 "ils.cpp"
|
|
|
|
|
|
|
|
|
|
|
|
da /c100 <addr> - ASCII string but this time show 100 chars per line
|
|
|
|
|
|
|
|
0:000> da /c100 xul!workerlz4_maxCompressedSize+0x9069b0
|
|
|
|
00000000`0ca6ac90 "/builds/worker/workspace/build/src/layout/svg/nsSVGIntegrationUtils.cpp"
|
|
|
|
```
|
|
|
|
|
|
|
|
At this point, we know what file and line we are dealing with. By comparing the callsites in the code with the 'call' instructions, we are almost always able to identify which function we have traversed into, until we finally have a decent idea what the crash is.
|
|
|
|
|
|
|
|
### Breaking where we want to break
|
|
|
|
|
|
|
|
The above section is good if you have a crash and want to go backwards. What if you want to break someone in particular, not related to a crash? Without symbols, how do you do it? There's a trick. Here's the code you'll want:
|
|
|
|
|
|
|
|
```plaintext
|
|
|
|
if(getenv("XXX_FINDME1"))
|
|
|
|
#ifdef __MINGW32__
|
|
|
|
__builtin_trap();
|
|
|
|
#else
|
|
|
|
DebugBreak();
|
|
|
|
#endif
|
|
|
|
```
|
|
|
|
|
|
|
|
This will let you break (conditionally, depending on the enviornment variable) right where you want.
|
|
|
|
|
|
|
|
You'll need to do the following dance:
|
|
|
|
|
|
|
|
1. Go into System -> Advanced System Settings -> Environment Variables and set your env var
|
|
|
|
2. THEN START WINDBG. This is very important. The Env Var change will not be picked up unless you start WinDBG after you set the variable (and close the dialog boxes I think)
|
|
|
|
3. Attach/Execute the program and get the address (as seen below)
|
|
|
|
4. CLOSE WINDBG
|
|
|
|
5. Delete the environment variable
|
|
|
|
6. REOPEN WINDBG
|
|
|
|
7. Attach/Execute the process and set your breakpoint
|
|
|
|
|
|
|
|
In MinGW the fault will look like this:
|
|
|
|
|
|
|
|
```plaintext
|
|
|
|
(3ec8.3a58): Illegal instruction - code c000001d (first chance)
|
|
|
|
(3ec8.3a58): Illegal instruction - code c000001d (!!! second chance !!!)
|
|
|
|
*** ERROR: Symbol file could not be found. Defaulted to export symbols for firefox.exe -
|
|
|
|
firefoxTargetSetOPMSigningKeyAndSequenceNumbers64+0x4086f:
|
|
|
|
00000000`0046490f 0f0b ud2
|
|
|
|
```
|
|
|
|
|
|
|
|
Disassemble that function, and the first line will give you the address of the function entry point, which you can then give to 'bp'
|
|
|
|
|
|
|
|
```plaintext
|
|
|
|
0:024> uf firefoxTargetSetOPMSigningKeyAndSequenceNumbers64+0x4086f
|
|
|
|
firefoxTargetSetOPMSigningKeyAndSequenceNumbers64+0x407f0:
|
|
|
|
```
|
|
|
|
|
|
|
|
If you're comparing to a non-MinGW build using DebugBreak(); the fault will look like the below. Note that I need to go one frame up in the call stack to get the correct function.
|
|
|
|
|
|
|
|
```plaintext
|
|
|
|
(7430.6d44): Break instruction exception - code 80000003 (first chance)
|
|
|
|
*** ERROR: Symbol file could not be found. Defaulted to export symbols for firefox.exe -
|
|
|
|
KERNELBASE!wil::details::DebugBreak+0x2:
|
|
|
|
00007ffe`42ce6482 cc int 3
|
|
|
|
0:023> k
|
|
|
|
# Child-SP RetAddr Call Site
|
|
|
|
00 00000060`bafff448 00007ff7`5ad7c436 KERNELBASE!wil::details::DebugBreak+0x2
|
|
|
|
01 00000060`bafff450 00007ff7`5ad996fe firefoxGetHandleVerifier+0x4d96
|
|
|
|
02 00000060`bafff4d0 00007ff7`5ad99ddb firefoxIsSandboxedProcess+0x3a0e
|
|
|
|
03 00000060`bafff790 00007ffe`45edc3bb firefoxIsSandboxedProcess+0x40eb
|
|
|
|
04 00000060`bafffa50 00007ffe`45e96be3 ntdllRtlpTpWaitCallback+0x9b
|
|
|
|
05 00000060`bafffac0 00007ffe`45e94c30 ntdllTppExecuteWaitCallback+0xab
|
|
|
|
06 00000060`bafffb00 00007ffe`45ac1fe4 ntdllTppWorkerThread+0x8d0
|
|
|
|
07 00000060`bafffe90 00007ffe`45ecf061 KERNEL32BaseThreadInitThunk+0x14
|
|
|
|
08 00000060`bafffec0 00000000`00000000 ntdllRtlUserThreadStart+0x21
|
|
|
|
0:023> uf firefoxGetHandleVerifier+0x4d96
|
|
|
|
firefoxGetHandleVerifier+0x4d40:
|
|
|
|
```
|
|
|
|
|
|
|
|
## A few other notes
|
|
|
|
|
|
|
|
If you see an address like xul!XRE_GetBootstrap+0x2508847 - that's a relative address (it's relative to xul.dll!XRE_GetBootstrap - It's not the _function_ XRE_GetBootstrap, it's 0x2508847 bytes after the start of that function). If you see an address like 00000000\`071da837 or 071da837 - that's an absolute addresses.
|
|
|
|
|
|
|
|
If a relative address points to code (it usually does), then that address will be the same across process executions. xul.dll will move around in memory due to ASLR, but as long as you're running the same build, the instructions at xul!XRE_GetBootstrap+0x2508847 will be the same.
|
|
|
|
|
|
|
|
The same is not true for absolute addresses. These addresses are not going to be the same across process execution.
|
|
|
|
|
|
|
|
## Symbols Appendix
|
|
|
|
|
|
|
|
Currently, building without stripping symbols for Windows produces a xul.dll over 2 GB. This is larger than is acceptable by Windows and thus it won't load the dll and firefox.exe will error very early on.
|
|
|
|
|
|
|
|
Using objcopy --add-gnu-debuglink one can move the debug sections to another file and tell gdb the symbols are there. This is explained in <https://stackoverflow.com/questions/866721/how-to-generate-gcc-debug-symbol-outside-the-build-target/866731#866731>
|
|
|
|
|
|
|
|
Once that it done, we can run firefox, and in theory tell gdb where the symbols are. However gdb choked on reading the symbols. This is detailed at <https://ritter.vg/misc/ff/dwarf-error.html> . Further investigation (in <https://ritter.vg/misc/ff/dwarf-error-2.html> ) revealed the culprit was yasm passing -dwarf2. The bypass for this is to comment out the -g dwarf2 line in <https://searchfox.org/mozilla-central/rev/08df4e6e11284186d477d7e5b0ae48483ecc979c/python/mozbuild/mozbuild/frontend/context.py#375> Once that is done, you will get a symbol file gdb can, supposedly, read.
|
|
|
|
|
|
|
|
All is not well. Following this fix, objcopy segfaulted: <https://sourceware.org/bugzilla/show_bug.cgi?id=23061> As explained in the bug, the suspicion is that something in gcc is overflowing (probably because it's a signed int and we're overflowing that) and resulting in corrupt DWARF information. At this point I ceased investigation.
|
|
|
|
|
|
|
|
Besides investigating the suspected gcc bug(s), there are other ways that are worth investigating for symbols:
|
|
|
|
|
|
|
|
- gz (not supported in MinGW??)
|
|
|
|
- gsplit-dwarf - This is supposed to move the dwarf data to separate files.
|
|
|
|
- g1 - This produces less debugging information. This produces a xul.dll that is within the size limits, but didn't run. The dwarf2 fix may be needed.
|
|
|
|
|
|
|
|
# mingw-clang
|
|
|
|
|
|
|
|
## Creating debug symbols
|
|
|
|
|
|
|
|
With Clang it is possible to create PDBs and use them to debug with WinDBG (and also with Visual Studio, to some extents).
|
|
|
|
|
|
|
|
You need to add `MOZ_COPY_PDBS=1` to `mozconfig-windows-x86_64`, however also some other options may be useful:
|
|
|
|
|
|
|
|
```
|
|
|
|
ac_add_options MOZ_COPY_PDBS=1
|
|
|
|
# This enables asserts
|
|
|
|
ac_add_options --enable-debug
|
|
|
|
# Make debug easier without optimizations
|
|
|
|
ac_add_options --disable-optimize
|
|
|
|
```
|
|
|
|
|
|
|
|
In this way, all the PDBs are created, but sadly not all are copied (`xul.pdb` is a notable exception; see #31546).
|
|
|
|
|
|
|
|
However, there is a workaround to get also the missing ones.
|
|
|
|
|
|
|
|
Add the following line after `./mach build --verbose` in `projects/firefox/build`:
|
|
|
|
|
|
|
|
```
|
|
|
|
find obj-mingw -path obj-mingw/dist -prune -o -name '*.pdb' -exec cp {} obj-mingw/dist/bin/ \;
|
|
|
|
```
|
|
|
|
|
|
|
|
Please notice that symbols are quite heavy (`xul.pdb` is almost 1GB alone) and their copy will make the creation of the installer quite longer.
|
|
|
|
|
|
|
|
Also notice that all references to the source code are absolute paths in the build environment. However if you open the source file manually, it is recognized without any problem.
|
|
|
|
|
|
|
|
## WinDBG Cheatsheet
|
|
|
|
|
|
|
|
There are three versions of WinDBG: WinDbg x86 WinDbg x64 WinDbg or WinDbg Preview
|
|
|
|
|
|
|
|
As far as I know, the commands in all of them are the same. But WinDbg Preview is the newest version, the easiest to install, and has a better interface for viewing multiple things at once. (That doesn't mean it's good, just that it's better.)
|
|
|
|
|
|
|
|
HOWEVER, the one thing that WinDbg Preview lacks is the Process and Threads window, which lets you switch which process you're debugging. That's important, so I usually use WinDbg x64/x86.
|
|
|
|
|
|
|
|
### Debugging Multi-Process Firefox
|
|
|
|
|
|
|
|
WinDbg Preview lets you automatically attach to all child processes. You'll hit a cc as a new process is launched and you can get breakpoints then.
|
|
|
|
|
|
|
|
In regular WinDbg, you can enable/disable the child process debugging setting by:
|
|
|
|
|
|
|
|
```plaintext
|
|
|
|
.childdbg 1 (or 0)
|
|
|
|
```
|
|
|
|
|
|
|
|
To switch processes, open the Processes and Threads window (which is not available in WinDbg Preview) and choose the process from that.
|
|
|
|
|
|
|
|
You can orient yourself when you land inside a process with the command
|
|
|
|
|
|
|
|
```plaintext
|
|
|
|
!peb
|
|
|
|
```
|
|
|
|
|
|
|
|
This will print the Process Environment Block, which will contain the PID, command line (good to telling what type of process you're in) and other information like Environment Variables.
|
|
|
|
|
|
|
|
If you're annoyed by the initial breakpoint on content process start, or the breakpoint on process end, you can disable them. This is especially useful if you've edited the source to include ::DebugBreak() where you want to break and thus don't need to set breakpoints.
|
|
|
|
|
|
|
|
```plaintext
|
|
|
|
sxi ibp - Disable the process starting breakpoint
|
|
|
|
sxi epr - Disable the process ending breakpoint
|
|
|
|
```
|
|
|
|
|
|
|
|
### WinDbg Commands
|
|
|
|
|
|
|
|
```plaintext
|
|
|
|
kn - call stack frames
|
|
|
|
.frame ## - switch the specified frame
|
|
|
|
```
|
|
|
|
|
|
|
|
```plaintext
|
|
|
|
d? <addr> - dump the memory at <addr> (choose of the below:)
|
|
|
|
db <addr> - dump the memory at <addr> as bytes
|
|
|
|
da <addr> - dump the memory at <addr> as an ASCII string
|
|
|
|
da /c100 <addr> - an ASCII string in 100 char columns
|
|
|
|
```
|
|
|
|
|
|
|
|
```plaintext
|
|
|
|
u <addr> - unassemble the memory at <addr>
|
|
|
|
u <addr> L10 - unassemble next 10 lines
|
|
|
|
uf <addr> - unassemble as function
|
|
|
|
```
|
|
|
|
|
|
|
|
You can work backwards by specify u -1 and so on. But when you do that, there is no guarentee you're unassembling on an instruction boundary, so you can wind up getting garbage instructions by beginning disassembly in the middle fo an instruction. (They'll show up as real assembly usually, but they don't make sense.) Because of the nature of assembly, after some smallish (5-15) number of instructions, they will almost already reconverge onto the correct assembly sequence. So try doing -x, -x+1, -x+2, and so on until you see something that seems like it's consistent.
|
|
|
|
|
|
|
|
uf is more reliable, it will disassemble the entire function block that is contained in. Untill it can't identify it as being inside a function, in which case it will fail and you'll have to use u.
|
|
|
|
|
|
|
|
```plaintext
|
|
|
|
bp <addr> - place a breakpoint at <addr>
|
|
|
|
```
|
|
|
|
|
|
|
|
There are variants of bp: bu and bm. They have to do with unresolved symbols or symbols matching a pattern. But I've never really had a problem just using bp - I think it gets converted to the correct type if it needs to be in most cases.
|
|
|
|
|
|
|
|
```plaintext
|
|
|
|
ba <r/w/e/i> <size> <addr>
|
|
|
|
```
|
|
|
|
|
|
|
|
Place a memory breakpoint at . Read or Write makes the most sense. is probably 4 or 8 bytes.
|
|
|
|
|
|
|
|
```plaintext
|
|
|
|
lm - print loaded modules
|
|
|
|
```
|
|
|
|
|
|
|
|
```plaintext
|
|
|
|
s -a <start> L?<Len> "string"
|
|
|
|
.foreach( addr {s -[1]a 0 L?FFFFFFFFFFFFFFFF "\Tom"}){da L100 addr}
|
|
|
|
```
|
|
|
|
|
|
|
|
search in memory for string. You'll need to specify a address. A good bet is to specify the start of the xul.dll module; unless of course the string you're lookign for is in another module, like mozglue. You can also specify 0 for the start; and 0xffffffffffffffff for the length. The ? in the command is supposed to be there, you need that. Additionally, the string (if it is a string and not a byte value) needs to be in double quotes).
|
|
|
|
|
|
|
|
You can loop over the results and dup them for easy skimming also.
|
|
|
|
|
|
|
|
```plaintext
|
|
|
|
p - single step execution (step over)
|
|
|
|
t - single step execution (step into)
|
|
|
|
pt - step to next return
|
|
|
|
pc - step to next call
|
|
|
|
```
|
|
|
|
|
|
|
|
In assembly mode, p will execution a single assembly instruction. When you have source code, it will execution a single line. I'm a little fuzzy on this, so experiment and correct as needed.
|
|
|
|
|
|
|
|
```plaintext
|
|
|
|
x xul!*symbolname* - Search for Symbols
|
|
|
|
```
|
|
|
|
|
|
|
|
If the address is relative to a function or module, then ASLR won't matter. If it's absolute, then it probably will.
|
|
|
|
|
|
|
|
```plaintext
|
|
|
|
dt <variablename> - displays the value of a variable
|
|
|
|
dv - dump all local variables
|
|
|
|
dt -v <variablename> - verbose information about a variable.
|
|
|
|
```
|
|
|
|
|
|
|
|
dt -v is useful when you need to get the address of a pointer, not the value of what it points to.
|
|
|
|
|
|
|
|
```plaintext
|
|
|
|
wt -oR
|
|
|
|
```
|
|
|
|
|
|
|
|
This is a complicated command that will execute the current function until its return AND print out all functions it calls (recursively) with the return value of them. If you have a Windows system call that's suceeding on one build but failing on another, this is useful to compare what those calls are doing and where they diverge.
|
|
|
|
|
|
|
|
### Conditional Breakpoints
|
|
|
|
|
|
|
|
Conditional breakpoints are fuckign ugly as sin. Here's the most common ones you'll need. Fortunately, if you get the syntax wrong, it will always break and tell you "Syntax error at "...
|
|
|
|
|
|
|
|
```plaintext
|
|
|
|
bp <address/function> "<complicated-condition-string>"
|
|
|
|
```
|
|
|
|
|
|
|
|
The 'gc' command you'll see all over the place is 'go from conditional breakpoint'. It means "Don't break". The absence of 'gc' means "break".
|
|
|
|
|
|
|
|
Checking member variables:
|
|
|
|
|
|
|
|
```plaintext
|
|
|
|
bp xul!mozilla::net::CacheFileMetadata::OnDataRead ".if(((@@c++(this->mBuf))==0x00)){.echo \"hit\"} .else{gc}"
|
|
|
|
bp xul!mozilla::net::CacheFileMetadata::GetElement ".if((@@c++(this->mBuf))==0x00 & (@@c++(this->mElementsSize))!=0x00) {.echo \"hit\"} .else{gc}"
|
|
|
|
```
|
|
|
|
|
|
|
|
Value in a register:
|
|
|
|
|
|
|
|
```plaintext
|
|
|
|
bp 06f480da "j @@rax = 0x80040111 '';'gc'"
|
|
|
|
```
|
|
|
|
|
|
|
|
Or a bitmask:
|
|
|
|
|
|
|
|
```plaintext
|
|
|
|
bp 00adff0f ".if (@eax & 0x01) = 0x0 {gc} .else {}"
|
|
|
|
```
|
|
|
|
|
|
|
|
Break on return value of function:
|
|
|
|
|
|
|
|
This is the same one as above (value in a register), except you specifically target the rax register (where the return value is stored) and you place the breakpoint in the right location (at the end of the function).
|
|
|
|
|
|
|
|
See <https://blogs.msdn.microsoft.com/jigarme/2007/10/28/how-to-break-in-windbg-when-particular-function-returns-specific-value/> Note the instructions there to get the return address actually get the first address after the function returns; which is specific to each callsite. It's better to use uf on the function in question and place a breakpoint on the ret instruction (of which there may be multiple).
|
|
|
|
|
|
|
|
Conditional breakpoint if a particular function is in the stack:
|
|
|
|
|
|
|
|
<https://stackoverflow.com/a/7800435>
|
|
|
|
|
|
|
|
That plus checking a return value:
|
|
|
|
|
|
|
|
```plaintext
|
|
|
|
bp 06f480da "r $t0 = 0;.foreach (v { k }) { .if ($spat(\"v\", \"*CacheFileMetadata::OnDataRead*\")) { r $t0 = 1;.break } }; .if( $t0 = 0 & rax = 0x80040111 ) { gc }"
|
|
|
|
```
|
|
|
|
|
|
|
|
^ that version is too slow. refactored to this version: /
|
|
|
|
|
|
|
|
```plaintext
|
|
|
|
bp 06f480da ".if( @rax = 0x80040111 ) { r $t0 = 0;.foreach (v { k }) { .if ($spat(\"v\", \"*CacheFileMetadata::OnDataRead*\")) { r $t0 = 1;.break } }; .if( $t0 = 0 ) { gc }}"
|
|
|
|
```
|
|
|
|
|
|
|
|
Conditional Breakpoint on String Comparison:
|
|
|
|
|
|
|
|
<https://docs.microsoft.com/en-us/windows-hardware/drivers/debugger/setting-a-conditional-breakpoint#span-idconditionalbreakpointbasedonstringcomparisonspanspan-idconditionalbreakpointbasedonstringcomparisonspanspan-idconditionalbreakpointbasedonstringcomparisonspanconditional-breakpoint-based-on-string-comparison>
|
|
|
|
|
|
|
|
## Debugging with Visual Studio
|
|
|
|
|
|
|
|
It is possible to debug also with Visual Studio, to some extents, even without the solution.
|
|
|
|
|
|
|
|
When you open VS, you can choose "Continue without code". At that point, you can attach to a process and debug it.
|
|
|
|
|
|
|
|
If you have debug symbols, in case of crashes, Visual Studio automatically asks for the path to the source file (since the one in the symbols is the absolute one in the build environment). Then it will recognize the source directory and load other files from there automatically.
|
|
|
|
|
|
|
|
If you need to set a breakpoint, you can set the source directory manually. Even though we do not have a VS solution, when you attach the exe, VS creates a temporary one. You can even modify its properties, where you can find «Debug Source Files». At that point you can add breakpoints even without crashes.
|
|
|
|
|
|
|
|
The only caveat is that I could not find a way to work with child processes. There is some extension to do that, but in my quick test, it did not work, while I could get the job done with WinDBG preview. |
|
|
|
\ No newline at end of file |
