Bug 1811487 - Clean-up popup hide / rollup APIs. r=cmartin

I'm about to extend them for bug 1811486, where I want to force in some
cases the rolled up popups to hide synchronously. These APIs use a ton
of boolean arguments that make them error prone, so refactor them a bit
to use strongly typed enums and flags.

Differential Revision: https://phabricator.services.mozilla.com/D167381