1. 17 May, 2020 1 commit
  2. 30 Apr, 2020 5 commits
  3. 28 Apr, 2020 1 commit
  4. 01 Apr, 2020 1 commit
  5. 31 Mar, 2020 1 commit
  6. 16 Jan, 2020 4 commits
  7. 15 Jan, 2020 5 commits
    • Karsten Loesing's avatar
      Bump version to 1.14.0-dev. · c75f0c78
      Karsten Loesing authored
      c75f0c78
    • Karsten Loesing's avatar
      Prepare for 1.14.0 release. · 3a9f05e0
      Karsten Loesing authored
      3a9f05e0
    • Karsten Loesing's avatar
      Update to metrics-lib 2.10.0. · 27e41ea7
      Karsten Loesing authored
      27e41ea7
    • Karsten Loesing's avatar
      Remember processed files between module runs. · 741401a0
      Karsten Loesing authored
      The three recently added modules to archive Snowflake statistics,
      bridge pool assignments, and BridgeDB metrics have in common that they
      process any input files regardless of whether they already processed
      them before.
      
      The problem is that the input files processed by these modules are
      either never removed (Snowflake statistics) or only removed manually
      by the operator (bridge pool assignments and BridgeDB statistics).
      
      The effect is that non-recent BridgeDB metrics and bridge pool
      assignments are being placed in the indexed/recent/ directory in the
      next execution after they are deleted for being older than 72 hours.
      The same would happen with Snowflake statistics after the operator
      removes them from the out/ directory.
      
      The fix is to use a state file containing file names of previously
      processed files and only process a file not found in there. This is
      the same approach as taken for bridge descriptor tarballs.
      741401a0
    • Karsten Loesing's avatar
      Update copyright to 2020. · d2a74b67
      Karsten Loesing authored
      d2a74b67
  8. 14 Jan, 2020 2 commits
    • Karsten Loesing's avatar
      Avoid reprocessing webstats files. · d4816337
      Karsten Loesing authored
      Web servers typically provide us with the last 14 days of request
      logs. We shouldn't process the whole 14 days over and over. Instead we
      should only process new logs files and any other log files containing
      log lines from newly written dates.
      
      In some cases web servers stop serving a given virtual host or stop
      acting as web server at all. However, in these cases we're left with
      14 days of logs per virtual host. Ideally, these logs would get
      cleaned up, but until that's the case, we should at least not
      reprocess these files over and over.
      
      In order to avoid reprocessing webstats files, we need a new state
      file with log dates contained in given input files. We use that state
      file to determine which of the previously processed webstats files to
      re-process, so that we can write complete daily logs.
      d4816337
    • Karsten Loesing's avatar
      3002d6bc
  9. 25 Nov, 2019 4 commits
  10. 22 Nov, 2019 3 commits
  11. 21 Nov, 2019 1 commit
  12. 20 Nov, 2019 7 commits
  13. 11 Nov, 2019 2 commits
  14. 09 Nov, 2019 3 commits