air gap the build machine
Here is the attack... An attacker finds out your build machines IP, buys a zero day exploit, gets access to the build machine, adds malicious code to the binary before it gets hashed and signed. To keep a low profile and to profit for a long time from the backdoor the exploit will only be used against selected high profile targets.
Since you don't have deterministic builds for everything (Tor, TBB) no one will find the backdoor. Don't expect people to thoroughly inspect each and every disassembly.
A good defense for network attacks against the build machine is using air gap.