dangerzone webdav processor merge requestshttps://gitlab.torproject.org/tpo/tpa/dangerzone-webdav-processor/-/merge_requests2023-09-25T19:10:57Zhttps://gitlab.torproject.org/tpo/tpa/dangerzone-webdav-processor/-/merge_requests/14Add .pdf file extension on safe files2023-09-25T19:10:57ZKezAdd .pdf file extension on safe filesCloses #22Closes #22https://gitlab.torproject.org/tpo/tpa/dangerzone-webdav-processor/-/merge_requests/13Add type signatures and mypy check2023-07-10T23:57:28ZKezAdd type signatures and mypy checkSee !10 for referenceSee !10 for referencehttps://gitlab.torproject.org/tpo/tpa/dangerzone-webdav-processor/-/merge_requests/12clarify a bit of obtuse logic2021-12-07T18:29:38Zanarcatclarify a bit of obtuse logicI was getting weird output from the script when trying to diagnose a
blocked dangerbot (e.g. #17 or #18). I couldn't figure out what was
going on and went to look at the code, then ended up staring at this
line for 2 minutes:
# ...I was getting weird output from the script when trying to diagnose a
blocked dangerbot (e.g. #17 or #18). I couldn't figure out what was
going on and went to look at the code, then ended up staring at this
line for 2 minutes:
# non-empty folder or regular file
if len(listing) > 1 or not path.endswith("/"):
It stuck with me. I couldn't actually remember (let alone figure out)
what that was supposed to do. I was so confused I actually ended up
re-proving De Morgan's Laws:
https://en.wikipedia.org/wiki/De_Morgan%27s_laws
The gist of the change is that we change this:
logging.info("sanitizing %s %s", folder, path)
# non-empty folder or regular file
if len(listing) > 1 or not path.endswith("/"):
# [...] do everything
To this:
if path.endswith("/") and len(listing) <= 1:
logging.info("skipping empty folder: %s/%s", folder, path)
return
logging.info("sanitizing %s %s", folder, path)
# [...] do everything
The diff looks huge because we reindent the entire "# [...] do
everything" block, but it's otherwise basically a noop, except we have
a better error message.
The output before:
```
anarcat@curie:dangerzone-webdav-processor(main)$ ./dangerzone-webdav-processor -l https://nc.torproject.net/remote.php/dav/files/dangerzone-bot/ -u dangerzone-bot -n -v
authenticated with webdav https://nc.torproject.net/remote.php/dav/files/dangerzone-bot/
sanitizing CVs (4)/ Candidate 33/
sanitizing CVs (4)/ Candidate 35/
```
after:
```
anarcat@curie:dangerzone-webdav-processor(main)$ ./dangerzone-webdav-processor -l https://nc.torproject.net/remote.php/dav/files/dangerzone-bot/ -u dangerzone-bot -n -v
authenticated with webdav https://nc.torproject.net/remote.php/dav/files/dangerzone-bot/
skipping empty folder: CVs (4)//Candidate 33/
skipping empty folder: CVs (4)//Candidate 35/
```anarcatanarcathttps://gitlab.torproject.org/tpo/tpa/dangerzone-webdav-processor/-/merge_requests/11blacken2021-12-07T18:28:29ZanarcatblackenThis should be a noop: just pass the entire file into `black` to have consistent style.
I've tried to keep it like this in the past, but it seems i failed at some point.This should be a noop: just pass the entire file into `black` to have consistent style.
I've tried to keep it like this in the past, but it seems i failed at some point.anarcatanarcathttps://gitlab.torproject.org/tpo/tpa/dangerzone-webdav-processor/-/merge_requests/10Add mypy type signatures2023-07-10T20:05:26ZKezAdd mypy type signaturesFollowing up on !8 again, this adds the mypy type signatures. They're a bit looser this time; no custom enums, ignoring things that would need a refactor to fix.Following up on !8 again, this adds the mypy type signatures. They're a bit looser this time; no custom enums, ignoring things that would need a refactor to fix.https://gitlab.torproject.org/tpo/tpa/dangerzone-webdav-processor/-/merge_requests/9Use os.path.join instead of string concatenation2021-07-08T15:18:46ZKezUse os.path.join instead of string concatenationClosing !8 in favor of short-lived feature branches with small changes.Closing !8 in favor of short-lived feature branches with small changes.https://gitlab.torproject.org/tpo/tpa/dangerzone-webdav-processor/-/merge_requests/8Modularize code to set up a proper rewrite2021-06-15T19:38:00ZKezModularize code to set up a proper rewritedangerzone-webdav-processor was renamed to `dangerzone_webdav_processor/__main__.py`, the remaining files were abstracted out of it
add script to run the dangerzone_webdav_processor module
create gitignore to ignore pip/mypy files
This ...dangerzone-webdav-processor was renamed to `dangerzone_webdav_processor/__main__.py`, the remaining files were abstracted out of it
add script to run the dangerzone_webdav_processor module
create gitignore to ignore pip/mypy files
This is just the start of the abstraction. I wanted to modularize things to make that easier first, and this should be merged back into the main branch so there aren't more changes to the old single-file script. I've tested (both the module and wrapper) and everything works fine On My Machineā¢https://gitlab.torproject.org/tpo/tpa/dangerzone-webdav-processor/-/merge_requests/7package this as something that can go into pip2021-06-10T19:52:07Zanarcatpackage this as something that can go into pipI'm not exactly sure we'll go that route, but it could make deployment
clearer.I'm not exactly sure we'll go that route, but it could make deployment
clearer.https://gitlab.torproject.org/tpo/tpa/dangerzone-webdav-processor/-/merge_requests/6explicitly upload only the processed path2021-06-10T19:07:17Zanarcatexplicitly upload only the processed pathThe `upload_sync()` function actually wipes the remote directory (or,
more precisely, removes files not present locally), so we actually
need to specify which component we need to upload, otherwise other
sanitized folders will get destro...The `upload_sync()` function actually wipes the remote directory (or,
more precisely, removes files not present locally), so we actually
need to specify which component we need to upload, otherwise other
sanitized folders will get destroyed.
Closes: #7https://gitlab.torproject.org/tpo/tpa/dangerzone-webdav-processor/-/merge_requests/5Upload each save file individually to fix issue 72021-06-10T20:16:33ZKezUpload each save file individually to fix issue 7The webdav library's `upload_sync` method was calling `upload_dir` which removes the directory if it already exists. this commit uploads each file individually with `os.walk` and `upload_async`.
Closes #7The webdav library's `upload_sync` method was calling `upload_dir` which removes the directory if it already exists. this commit uploads each file individually with `os.walk` and `upload_async`.
Closes #7https://gitlab.torproject.org/tpo/tpa/dangerzone-webdav-processor/-/merge_requests/4document a systemd deployment2021-06-10T19:14:12Zanarcatdocument a systemd deploymentThis hasn't actually been tested but, really, "how hard can this be?"
Once concern is that systemd might not take care of locking and that a
job might take more than one hour and run in parallel. We're pretty
good at avoiding race condi...This hasn't actually been tested but, really, "how hard can this be?"
Once concern is that systemd might not take care of locking and that a
job might take more than one hour and run in parallel. We're pretty
good at avoiding race conditions (by moving stuff to
`dangerzone/processing`, mostly) but we definitely should avoid two
processors in parallel. So we should make sure that only one instance
can be started by systemd, which is unclear to me right now.
Closes: #1https://gitlab.torproject.org/tpo/tpa/dangerzone-webdav-processor/-/merge_requests/3handle already existing remote files on move2021-06-10T18:19:09Zanarcathandle already existing remote files on moveFor some reason, we have files in the processing folder that are
already present before moving. Skip those files instead of crashing
with:
```
webdav3.exceptions.ResponseErrorCode: Request to https://nc.torproject.net/remote.php/dav/fil...For some reason, we have files in the processing folder that are
already present before moving. Skip those files instead of crashing
with:
```
webdav3.exceptions.ResponseErrorCode: Request to https://nc.torproject.net/remote.php/dav/files/dangerzone-bot/CVS/dangerzone-processing/2/ failed with code 412 and message: b'<?xml version="1.0" encoding="utf-8"?>\n<d:error xmlns:d="DAV:" xmlns:s="http://sabredav.org/ns">\n <s:exception>Sabre\\DAV\\Exception\\PreconditionFailed</s:exception>\n <s:message>The destination node already exists, and the overwrite header is set to false</s:message>\n <s:header>Overwrite</s:header>\n</d:error>\n'
```https://gitlab.torproject.org/tpo/tpa/dangerzone-webdav-processor/-/merge_requests/2hide all dangerous folders under a single dangerzone/ folder2021-06-10T18:19:02Zanarcathide all dangerous folders under a single dangerzone/ folderRight now we have `dangerzene-processed/`, `dangerzone-rejected/`,
`dangerzone-processing/`... that's noisy, at best, and dangerous, at
worst: at least one user thought that `dangerzone-processed` was where
processed (so: safe) files end...Right now we have `dangerzene-processed/`, `dangerzone-rejected/`,
`dangerzone-processing/`... that's noisy, at best, and dangerous, at
worst: at least one user thought that `dangerzone-processed` was where
processed (so: safe) files ended up.
Just have `dangerzone/` and `safe/` folders at the toplevel, with the
`processed/`, `rejected/`, `processing/` stuff inside `dangerzone/`.
Closes: #4https://gitlab.torproject.org/tpo/tpa/dangerzone-webdav-processor/-/merge_requests/1fix incorrect SPECIAL_FOLDERS path check2021-06-10T17:00:12ZKezfix incorrect SPECIAL_FOLDERS path checkremove trailing slash from `SPECIAL_FOLDERS` entry
create function to normalize paths before checking to make things easier for caller
closes #8remove trailing slash from `SPECIAL_FOLDERS` entry
create function to normalize paths before checking to make things easier for caller
closes #8