Skip to content
Snippets Groups Projects

CollecTor stopped collecting snowflake stats

    • Closed (moved) Issue created by Cecylia Bocovich @cohosh

    Looks like this coincides with when the bamsoftware.com and freehaven.net domains were switched to the new host (#29258 (moved)).

    Perhaps CollecTor needs to update it's DNS cache? In any case, it's safe to point to snowflake-broker.torproject.net again.

    To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information

    Child items 0

    • No child items are currently assigned. Use child items to break down this issue into smaller parts.

    Activity

    • Cecylia Bocovich added component::metrics/collector owner::metrics-team priority::medium resolution::fixed severity::normal snowflake status::closed type::defect labels

      added component::metrics/collector owner::metrics-team priority::medium resolution::fixed severity::normal snowflake status::closed type::defect labels

    • Karsten Loesing
      Karsten Loesing @karsten ·

      I did notice this problem, too, but did not get to letting you know about it yet.

      So, when I open https://snowflake-broker.freehaven.net/metrics or https://snowflake-broker.torproject.net/metrics in a browser, I see outdated statistics with all zeroes. That's what CollecTor sees, too.

      Could it be that you need to update something on your side?

      Or can you tell me a URL that has recent statistics that I can put into CollecTor's config file? Thanks!

    • Cecylia Bocovich
      Cecylia Bocovich @cohosh ·
      Author

      Hmm, I'm also getting outdated stats when I manually fetch the metrics log. This appears to be a snowflake issue. It's weird because the updated log file is there, the broker just doesn't appear to be serving it. I've created #32576 (moved).

    • Cecylia Bocovich
      Cecylia Bocovich @cohosh ·
      Author

      Okay I copied the old snowflake broker logs to the new host just so we have data for the last week.

      I'm still not sure why the new broker host hasn't been logging metrics data, I fear we have lost data for the new host since the 14th, but CollecTor should be able to get more data now.

    • Cecylia Bocovich
      Cecylia Bocovich @cohosh ·
      Author

      Found the problem, there's a bug in the broker that's causing it to crash. We don't collect metrics if it crashes so that explains why the new broker wasn't updating the metrics log file.

      Anyway, CollecTor should atleast be able to get data for the last week now since I moved the log from the old host to the new one.

      Trac:
      Status: new to needs_review

    • Karsten Loesing
      Karsten Loesing @karsten ·

      Ah, I noticed just now that this ticket needs action again.

      So, I looked at the files that CollecTor archived in the past 4-5 weeks. Can you tell me which of them I should delete? Some cases seem obvious (snowflake-ips-total 0), but some are less obvious (to me). Here are the files in question:

      2019-10-26-14-29-08-snowflake-stats:snowflake-ips-total 4158
2019-10-27-14-29-08-snowflake-stats:snowflake-ips-total 3772
2019-10-28-14-29-08-snowflake-stats:snowflake-ips-total 4310
2019-10-29-14-29-08-snowflake-stats:snowflake-ips-total 4925
2019-10-30-14-29-08-snowflake-stats:snowflake-ips-total 4837
2019-10-31-14-29-08-snowflake-stats:snowflake-ips-total 4676
2019-11-01-14-29-08-snowflake-stats:snowflake-ips-total 4467
2019-11-01-19-44-19-snowflake-stats:snowflake-ips-total 0
2019-11-02-14-29-08-snowflake-stats:snowflake-ips-total 4332
2019-11-02-19-44-19-snowflake-stats:snowflake-ips-total 0
2019-11-03-14-29-08-snowflake-stats:snowflake-ips-total 3917
2019-11-03-19-44-19-snowflake-stats:snowflake-ips-total 0
2019-11-04-14-29-08-snowflake-stats:snowflake-ips-total 4294
2019-11-04-19-44-19-snowflake-stats:snowflake-ips-total 0
2019-11-05-14-29-08-snowflake-stats:snowflake-ips-total 4769
2019-11-05-19-44-19-snowflake-stats:snowflake-ips-total 0
2019-11-06-14-29-08-snowflake-stats:snowflake-ips-total 4729
2019-11-06-19-44-19-snowflake-stats:snowflake-ips-total 0
2019-11-07-14-29-08-snowflake-stats:snowflake-ips-total 4708
2019-11-07-19-44-19-snowflake-stats:snowflake-ips-total 0
2019-11-08-14-29-08-snowflake-stats:snowflake-ips-total 4538
2019-11-08-19-44-19-snowflake-stats:snowflake-ips-total 0
2019-11-09-14-29-08-snowflake-stats:snowflake-ips-total 4277
2019-11-09-19-44-19-snowflake-stats:snowflake-ips-total 0
2019-11-10-14-29-08-snowflake-stats:snowflake-ips-total 3788
2019-11-10-19-44-19-snowflake-stats:snowflake-ips-total 0
2019-11-11-14-29-08-snowflake-stats:snowflake-ips-total 4331
2019-11-11-19-44-19-snowflake-stats:snowflake-ips-total 0
2019-11-12-14-29-08-snowflake-stats:snowflake-ips-total 4669
2019-11-12-19-44-19-snowflake-stats:snowflake-ips-total 0
2019-11-13-14-29-08-snowflake-stats:snowflake-ips-total 4609
2019-11-13-19-44-19-snowflake-stats:snowflake-ips-total 0
2019-11-14-19-44-19-snowflake-stats:snowflake-ips-total 0
2019-11-14-20-42-57-snowflake-stats:snowflake-ips-total 3500
2019-11-15-20-42-57-snowflake-stats:snowflake-ips-total 1667
2019-11-16-20-42-57-snowflake-stats:snowflake-ips-total 80
2019-11-17-20-42-57-snowflake-stats:snowflake-ips-total 20
2019-11-18-20-42-57-snowflake-stats:snowflake-ips-total 4
2019-11-19-20-42-57-snowflake-stats:snowflake-ips-total 2
2019-11-20-20-42-57-snowflake-stats:snowflake-ips-total 1
2019-11-21-20-42-57-snowflake-stats:snowflake-ips-total 1
2019-11-24-08-24-18-snowflake-stats:snowflake-ips-total 3915
2019-11-25-08-24-18-snowflake-stats:snowflake-ips-total 4182
2019-11-26-08-24-18-snowflake-stats:snowflake-ips-total 4237
2019-11-27-08-24-18-snowflake-stats:snowflake-ips-total 4344

      If you could give me a list of files to delete, that would be great!

      Trac:
      Status: needs_review to needs_information

    • Cecylia Bocovich
      Cecylia Bocovich @cohosh ·
      Author

      Let's only delete the files with snowflake-ips-total. I want to keep the others around since they have useful data about the broker migration.

    • Karsten Loesing
      Karsten Loesing @karsten ·

      Replying to cohosh:

      Let's only delete the files with snowflake-ips-total.

      Hmm? Do you mean we should delete all files with "snowflake-ips-total 0", or something else?

    • Cecylia Bocovich
      Cecylia Bocovich @cohosh ·
      Author

      Replying to karsten:

      Replying to cohosh:

      Let's only delete the files with snowflake-ips-total.

      Hmm? Do you mean we should delete all files with "snowflake-ips-total 0", or something else?

      Whoops. Yes, that's what I meant.

    • Karsten Loesing
      Karsten Loesing @karsten ·

      Done. Will check tarballs in a few days to confirm that these files are actually gone.

    • Karsten Loesing
      Karsten Loesing @karsten ·

      Updated tarballs no longer contain deleted files. Resolving.

      Trac:
      Status: needs_information to closed
      Resolution: N/A to fixed

    • Trac closed

      closed

    • Cecylia Bocovich mentioned in issue #32576 (moved)

      mentioned in issue #32576 (moved)

    Please register or sign in to reply