Data files are not written atomically

For example, when running out of disk space, with this traceback:

Exception in thread Thread-1:
OSError: [Errno 28] No space left on device
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/lib/python3.7/threading.py", line 917, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.7/threading.py", line 865, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.7/dist-packages/sbws-1.1.0+228.gdbefc46c-py3.7.egg/sbws/lib/resultdump.py", line 843, in enter
    self.handle_result(r)
  File "/usr/local/lib/python3.7/dist-packages/sbws-1.1.0+228.gdbefc46c-py3.7.egg/sbws/lib/resultdump.py", line 784, in handle_result
    write_result_to_datadir(result, self.datadir)
  File "/usr/local/lib/python3.7/dist-packages/sbws-1.1.0+228.gdbefc46c-py3.7.egg/sbws/lib/resultdump.py", line 185, in write_result_to_datadir
    fd.write('{}\n'.format(str(result)))
OSError: [Errno 28] No space left on device

Next time it tries to read the "results" in the datadir or the state.dat file, the json is malformed:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/sbws-1.1.0+228.gdbefc46c-py3.7.egg/sbws/core/scanner.py", line 771, in run_speedtest
    main_loop(args, conf, controller, rl, cb, rd, rp, destinations, pool)
  File "/usr/local/lib/python3.7/dist-packages/sbws-1.1.0+228.gdbefc46c-py3.7.egg/sbws/core/scanner.py", line 622, in main_loop
    hbeat.print_heartbeat_message()
  File "/usr/local/lib/python3.7/dist-packages/sbws-1.1.0+228.gdbefc46c-py3.7.egg/sbws/lib/heartbeat.py", line 48, in print_heartbeat_message
    loops_count = self.state_dict.count('recent_priority_list')
  File "/usr/local/lib/python3.7/dist-packages/sbws-1.1.0+228.gdbefc46c-py3.7.egg/sbws/util/state.py", line 103, in count
    if self.get(k):
  File "/usr/local/lib/python3.7/dist-packages/sbws-1.1.0+228.gdbefc46c-py3.7.egg/sbws/util/state.py", line 69, in get
    self._state = self._read()
  File "/usr/local/lib/python3.7/dist-packages/sbws-1.1.0+228.gdbefc46c-py3.7.egg/sbws/util/state.py", line 53, in _read
    return json.load(fd, cls=CustomDecoder)
  File "/usr/lib/python3.7/json/__init__.py", line 296, in load
    parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
  File "/usr/lib/python3.7/json/__init__.py", line 361, in loads
    return cls(**kw).decode(s)
  File "/usr/local/lib/python3.7/dist-packages/sbws-1.1.0+228.gdbefc46c-py3.7.egg/sbws/util/json.py", line 24, in decode
    decoded = super().decode(s, **kwargs)
  File "/usr/lib/python3.7/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.7/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

At least, it exits all clean it cans:

Call stack:
  File "/usr/local/bin/sbws", line 11, in <module>
    load_entry_point('sbws==1.1.0+228.gdbefc46c', 'console_scripts', 'sbws')()
  File "/usr/local/lib/python3.7/dist-packages/sbws-1.1.0+228.gdbefc46c-py3.7.egg/sbws/sbws.py", line 75, in main
    exit(comm['f'](*comm['a'], **comm['kw']))
  File "/usr/local/lib/python3.7/dist-packages/sbws-1.1.0+228.gdbefc46c-py3.7.egg/sbws/core/scanner.py", line 810, in main
    run_speedtest(args, conf)
  File "/usr/local/lib/python3.7/dist-packages/sbws-1.1.0+228.gdbefc46c-py3.7.egg/sbws/core/scanner.py", line 780, in run_speedtest
    stop_threads(signal.SIGTERM, None, 1)
  File "/usr/local/lib/python3.7/dist-packages/sbws-1.1.0+228.gdbefc46c-py3.7.egg/sbws/core/scanner.py", line 53, in stop_threads
    log.debug('Stopping sbws.')
Message: 'Stopping sbws.'
Arguments: ()
sbws.service: Main process exited, code=exited, status=1/FAILURE
sbws.service: Failed with result 'exit-code'.

This also caused #40020 (closed).

While we don't implement #40024 (closed) (most dbms handle atomic operations), we might want to use something like https://github.com/untitaker/python-atomicwrites.