Lightly refactor and test format_hex_number_sigsafe

Better tests for upper bounds, and for failing cases.

Also, change the function's interface to take a buffer length rather
than a maximum length, and then NUL-terminate: functions that don't
NUL-terminate are trouble waiting to happen.