In legacy/trac#20499 (moved) we see a potential bug where relays opt to stop getting directory information.
How pervasive is this bug in practice?
Doing a scan of these relays to see which consensus they give us could be quite enlightening.
Also, once we have a module in our scanners to do this test, it would probably be wise to automate running it every so often, as an early (ok, not so early really) warning system to catch regressions.
(In theory all 0.2.9 relays should be caching directory stuff now, right?)
Designs
Child items
...
Show closed items
Linked items
0
Link issues together to show that they're related.
Learn more.
(I think all of the relays cache dir stuff, and serve it over their ORPort via begindir, but not all of them will have an open dirport, and I think atagar might find it easier to do the test via their dirport. Looking first at relays that are easy to test is fine by me.)
Hi Roger, seems 0.3.0.0 is having issues. 20 of 5002 relays with a DirPort are serving stale consensus data. In total there's 22 relays running 0.3.0.0-alpha-dev so this is just about all of them..
FC64CD763F8C1A319BFBBF62551684F4E1E42332 (0.3.0.0-alpha-dev): consensus expired at 2016-10-28T16:00:00F864BDEB66C8A11E209456846499C2F995435797 (0.3.0.0-alpha-dev): consensus expired at 2016-10-24T07:00:00EF887B1997DD786E47CBCAA3F445212467356EBE (0.3.0.0-alpha-dev): consensus expired at 2016-10-22T03:00:00EF65A4EFF948EA429430FDB834AB9C5F992E81F6 (0.3.0.0-alpha-dev): consensus expired at 2016-10-28T16:00:00D5B8C38539C509380767D4DE20DE84CF84EE8299 (0.2.9.4-alpha): consensus expired at 2016-10-21T16:00:00CCC158EA3BB02E4E443BD1F317457CC9E845B050 (0.3.0.0-alpha-dev): consensus expired at 2016-10-28T15:00:00C79552275DFCD486B942510EF663ED36ACA1A84B (0.3.0.0-alpha-dev): consensus expired at 2016-10-27T12:00:00B210DF76804619395044D139E826D4AAC939CADB (0.3.0.0-alpha-dev): consensus expired at 2016-10-28T15:00:00ABF5C38A93F2D7E77A226871AB0ADB052279B48F (0.3.0.0-alpha-dev): consensus expired at 2016-10-30T13:00:00A6B0521C4C1FB91FB66398AAD523AD773E82E77E (0.3.0.0-alpha-dev): consensus expired at 2016-10-28T03:00:00A69221A7EC7498D2F88A0FB795261013FA36CAAE (0.3.0.0-alpha-dev): consensus expired at 2016-10-26T23:00:009E0B9F902F83E96C352E3B9ECEA4B60B1301A7A1 (0.3.0.0-alpha-dev): consensus expired at 2016-10-20T03:00:007FF5DF2E673F4F171C6DB99BA0FA2D9F32A17152 (0.2.9.4-alpha): consensus expired at 2016-10-24T22:00:0051377C496818552E263583A44C796DF3FB0BC71B (0.3.0.0-alpha-dev): consensus expired at 2016-10-28T02:00:0048288B0E2473F54A8B9029E0231F2D2F7F7967FD (0.3.0.0-alpha-dev): consensus expired at 2016-10-26T22:00:0041F07731207742860D43AC426FBAE2F3947BD1CA (0.3.0.0-alpha-dev): consensus expired at 2016-10-26T16:00:003DE567C1350C0E858C6147AECB06EA9B3EAF3261 (0.2.9.2-alpha): consensus expired at 2016-10-13T07:00:0038A42B8D7C0E6346F4A4821617740AEE86EA885B (0.3.0.0-alpha-dev): consensus expired at 2016-10-28T02:00:002D49BA84E74E373CD73CDEB09227FDF116155A07 (0.3.0.0-alpha-dev): consensus expired at 2016-10-28T15:00:00198E72D653B790F7BED7247812C1F96F7CCE56FF (0.3.0.0-alpha-dev): consensus expired at 2016-10-27T02:00:00
Full results are attached to this ticket. Here's the script...
Also, once we have a module in our scanners to do this test
This script took 639 minutes to run (yikes!). Downloading 5002 consensuses is a non-trivial load on the network so not something I think we should run on an ongoing fashion unless we have reason to suspect a regression.
I think all of the relays cache dir stuff, and serve it over their ORPort...
This requires teaching Stem to talk with tor's ORPort (which I'd love!). Yawning thought this wouldn't be hard to do.
Would you be able to repeat the scan for the microdesc consensuses? (armadev told me on #tor-dev to ask). There are definitely relays that have a perfectly valid normal consensus but a stale microdesc consensus, and I think the microdesc consensus being out of date is a more prevalent issue (and I'm told a bigger issue because more clients use them).
The best (worst?) example I found (just checking random ones from the consensus) is this one:
Spec updated and stem can fetch them. Only change to the above script was to add 'microdescriptor = True' to the get_consensus() call. 0.2.9.4 certainly does have an issue...
% grep 'consensus expired' /home/atagar/results_micro.txtE64DAA637261410A49C0785E37A100D5BF0D4E1E (0.2.9.4-alpha): consensus expired at 2016-11-02T17:00:00A9D1F803E1E21094D6EC643909FD033D805ADA88 (0.2.9.4-alpha): consensus expired at 2016-11-01T15:00:00ED6E43F07ABE87C017BD80D4BA24E41F8FF32E94 (0.2.9.4-alpha): consensus expired at 2016-11-01T13:00:00E2D82421FDF9F17D02D2E640BF31F3E650D585CF (0.2.9.4-alpha): consensus expired at 2016-11-01T09:00:00FC9AC8EA0160D88BCCFDE066940D7DD9FA45495B (0.2.9.4-alpha): consensus expired at 2016-11-03T09:00:00FC5DABC0086E959673F32B76EB46EAA2E7F06A87 (0.2.9.4-alpha): consensus expired at 2016-11-03T08:00:00FBA48003F44D7C623ADA8CE46AA6EAEDDF851AD3 (0.2.9.4-alpha): consensus expired at 2016-11-03T08:00:00F833F100DA6A7B4DB58CC171B22B0A27DBC30321 (0.2.9.2-alpha-dev): consensus expired at 2016-11-02T02:00:00F409FA7902FD89270E8DE0D7977EA23BC38E5887 (0.2.9.4-alpha): consensus expired at 2016-11-03T09:00:00F00EC2E0A2CA79A57FE7A0918A087987747D772D (0.2.9.4-alpha): consensus expired at 2016-10-26T03:00:00EF812C2498DE0ABB5367C30E8544F668C464517C (0.2.9.4-alpha): consensus expired at 2016-11-03T14:00:00ED4B0DBA79AEF5521564FA0231455DCFDDE73BB6 (0.2.9.3-alpha): consensus expired at 2016-11-03T09:00:00EC639EDAA5121B47DBDF3D6B01A22E48A8CB6CC7 (0.2.9.4-alpha): consensus expired at 2016-11-02T17:00:0017CC33287E23F4B0CD8B0C947B48E437AFCA9B2C (0.2.9.4-alpha): consensus expired at 2016-11-03T07:00:00E747773C61CF671A222EB913DE17DA33074A6D76 (0.2.9.4-alpha): consensus expired at 2016-10-18T20:00:00E431079AB7766ACE208442474D7E4009932432AA (0.2.9.3-alpha): consensus expired at 2016-11-02T17:00:00E382042E06A0A68AFC533E5AD5FB6867A12DF9FF (0.2.9.4-alpha): consensus expired at 2016-10-30T12:00:00DEE5298B3BA18CDE651421CD2DCB34A4A69F224D (0.2.9.4-alpha): consensus expired at 2016-11-03T08:00:00DE134FC8E5CC4EC8A5DE66934E70AC9D70267197 (0.2.9.2-alpha): consensus expired at 2016-09-20T15:00:00D97E20FE8F445A7C8F1E9556B1E6AD9B00CD05B7 (0.2.9.2-alpha): consensus expired at 2016-10-26T01:00:00D8C050523145AC33F37B58695319BAAF0B8884A7 (0.2.9.3-alpha): consensus expired at 2016-10-31T11:00:00D7DB8E82604F806766FC3F80213CF719A0481D0B (0.2.9.2-alpha): consensus expired at 2016-10-15T01:00:00D5B8C38539C509380767D4DE20DE84CF84EE8299 (0.2.9.4-alpha): consensus expired at 2016-10-21T15:00:00D46175487C332CC602F7D1A44D7BA73348C6C49E (0.2.9.3-alpha): consensus expired at 2016-10-06T02:00:00D3A947CDB219DC9F7DB0B28F6E503D26AA1DCB35 (0.2.9.3-alpha): consensus expired at 2016-10-12T09:00:00CFBBA0D858F02E40B1432A65F6D13C9BDFE7A46B (0.2.9.3-alpha-dev): consensus expired at 2016-10-17T19:00:00CEF0D7CB4A8619C4E5D0F75FFFB30961469D3E6D (0.2.9.4-alpha): consensus expired at 2016-10-29T03:00:00CE0584F8FF2AAACB28E233AD380CA5C386CAF645 (0.2.9.4-alpha): consensus expired at 2016-11-01T21:00:00CD87B5554B4FFCE6038A4508794127219766C6F5 (0.2.7.6): consensus expired at 2016-11-01T09:00:00CBC6EB0E7299FB2A84FCF61AED02D4458291AA90 (0.2.9.4-alpha): consensus expired at 2016-10-28T17:00:00C9B68C802CA20C3E4FA46D77153D6EDC80F13CF5 (0.2.9.4-alpha): consensus expired at 2016-11-03T03:00:00C9933B3725239B6FAB5227BA33B30BE7B48BB485 (0.2.9.3-alpha): consensus expired at 2016-11-03T09:00:004B084AD6A0BA70761A333829F52042BB6EA009AF (0.2.9.4-alpha): consensus expired at 2016-11-02T05:00:00C818E0A91101ED0B302557089729C585B53292E3 (0.2.9.4-alpha): consensus expired at 2016-11-03T14:00:00C804BE8FB1C7C42D43C4A5E2039E77AA0FF3A8B4 (0.2.9.2-alpha): consensus expired at 2016-10-15T11:00:00C6DC18B9700A2B8A11071A0050BB21BF751C379D (0.2.9.4-alpha): consensus expired at 2016-11-02T05:00:00BC630CBBB518BE7E9F4E09712AB0269E9DC7D626 (0.2.9.4-alpha): consensus expired at 2016-11-03T00:00:00B7EC0C02D7D9F1E31B0C251A6B058880778A0CD1 (0.2.9.2-alpha): consensus expired at 2016-08-26T12:00:00B70629C29A032979E5AD00CC51D2330845AAFD6A (0.2.9.4-alpha): consensus expired at 2016-10-27T03:00:00B0B29389B594841812B4282F6C939035C60D5300 (0.2.9.4-alpha): consensus expired at 2016-11-02T20:00:00B01B7973207B1FD2F71D51D1B51B89EE77035CF4 (0.2.9.2-alpha): consensus expired at 2016-09-09T13:00:00AED76373324653A0522DF30550BA31902B2CFA44 (0.2.9.4-alpha): consensus expired at 2016-10-28T03:00:00AE55CEC84B700FEF3FB40D5ADAEECBBC10F60D15 (0.2.9.3-alpha): consensus expired at 2016-10-18T09:00:00AD2DD2C1A293A0DB01330B5E5DA5E37B946A42FA (0.2.9.4-alpha): consensus expired at 2016-10-29T20:00:00ACDC459DA2B1237289DDD713680682DAF393A187 (0.2.9.4-alpha): consensus expired at 2016-10-27T00:00:00ACD889D86E02EDDAB1AFD81F598C0936238DC6D0 (0.2.9.4-alpha): consensus expired at 2016-11-03T09:00:00ABCB4965F1FEE193602B50A365425105C889D3F8 (0.2.9.4-alpha): consensus expired at 2016-11-03T14:00:00A6DD1938D3D6FD03B5F1F96A79C1651291FE636D (0.2.9.4-alpha): consensus expired at 2016-10-29T03:00:00A556D1F290B643E6B6E6B2D119F041308D2E9C26 (0.2.9.4-alpha): consensus expired at 2016-11-03T17:00:00A4928A5E9544D814522B7BC8DAEA513E099F6514 (0.2.9.4-alpha-dev): consensus expired at 2016-10-27T22:00:00A40E1C039224FA8072C7C84F729236FD738C69DA (0.2.9.4-alpha): consensus expired at 2016-11-02T01:00:009FBEB75E8BC142565F12CBBE078D63310236A334 (0.2.9.4-alpha): consensus expired at 2016-11-02T06:00:009DA76A2AE6D08900B92D7F59E37814C7CE3056E5 (0.2.9.3-alpha-dev): consensus expired at 2016-10-13T16:00:009A68B85A02318F4E7E87F2828039FBD5D75B0142 (0.2.9.4-alpha): consensus expired at 2016-11-03T00:00:009844B981A80B3E4B50897098E2D65167E6AEF127 (0.2.9.3-alpha-dev): consensus expired at 2016-10-17T19:00:009712386010B1E579E750CEF612BD084DA4AAC1E0 (0.2.9.2-alpha-dev): consensus expired at 2016-09-15T16:00:009285B22F7953D7874604EEE2B470609AD81C74E9 (0.2.9.3-alpha-dev): consensus expired at 2016-10-17T19:00:009030DCF419F6E2FBF84F63CBACBA0097B06F557E (0.2.9.2-alpha): consensus expired at 2016-08-26T12:00:008FF2E7B7BF4C121D1F49D6606987C5802DDB9577 (0.2.9.4-alpha): consensus expired at 2016-10-26T12:00:00896364B7996F5DFBA0E15D1A2E06D0B98B555DD6 (0.2.9.4-alpha): consensus expired at 2016-10-28T01:00:0088D5F0BD87F9BC1A96FCBFDCEDB1C0E6BAB1D14E (0.2.9.4-alpha): consensus expired at 2016-11-01T23:00:008096552438C9B8AB82BDCB8D5896128B61AAFAF9 (0.2.9.4-alpha): consensus expired at 2016-11-03T14:00:007FF5DF2E673F4F171C6DB99BA0FA2D9F32A17152 (0.2.9.4-alpha): consensus expired at 2016-10-19T12:00:007F1D94E2C36F8CC595C2AB00022A5AE38171D50B (0.2.9.4-alpha): consensus expired at 2016-10-30T12:00:007BB160A8F54BD74F3DA5F2CE701E8772B841859D (0.2.9.3-alpha): consensus expired at 2016-09-28T22:00:007B8B083157372DB2994FEB5B075BBAE22BA2DC4F (0.2.9.4-alpha): consensus expired at 2016-10-31T16:00:007796D80C962044E4AEAEE94BACDA2E152591F6C5 (0.2.9.4-alpha): consensus expired at 2016-10-27T00:00:006DE61A6F72C1E5418A66BFED80DFB63E4C77668F (0.2.9.4-alpha): consensus expired at 2016-11-02T06:00:0068CECB9ECDF5EFE8FF23C25B926D756FB346FD92 (0.2.9.4-alpha): consensus expired at 2016-11-02T05:00:005FE7E83A85BFEF7BB2894D9483C9754DC1052067 (0.2.9.4-alpha): consensus expired at 2016-11-03T16:00:005EB8D862E70981B8690DEDEF546789E26AB2BD24 (0.2.9.3-alpha): consensus expired at 2016-09-28T22:00:005CE3AD8AD04ADE66C0037A3CF5F7F7A40D48A20B (0.2.9.4-alpha): consensus expired at 2016-10-23T19:00:0058DCF0E6048D1003BFD0CFF2D0F4E9E1F75A96ED (0.2.9.4-alpha): consensus expired at 2016-10-26T05:00:004E021F36F95EDD4F93F824020E3C5BFEA7B595CC (0.2.9.3-alpha): consensus expired at 2016-10-20T15:00:004888770464F0E900EFEF1BA181EA873D13F7713C (0.2.9.3-alpha): consensus expired at 2016-09-28T22:00:003DE567C1350C0E858C6147AECB06EA9B3EAF3261 (0.2.9.2-alpha): consensus expired at 2016-08-25T15:00:003B4FA23831CC69C136418B0305381A5103A3B470 (0.2.9.4-alpha): consensus expired at 2016-11-04T03:00:003A720B6E420E6A8133A0B7A4C37C08B9C9DEE5FD (0.2.9.2-alpha-dev): consensus expired at 2016-09-16T01:00:0039B6A1DE99D2DD649632CABB6C72827664E3CB29 (0.2.9.4-alpha): consensus expired at 2016-11-04T04:00:003711E80B5B04494C971FB0459D4209AB7F2EA799 (0.2.9.3-alpha-dev): consensus expired at 2016-10-17T19:00:0035442E6538EEEC0CC34C23C1FFB6045DCDA22128 (0.2.9.4-alpha): consensus expired at 2016-11-03T17:00:0032828476F4F84E15C42B4C360A5CD8DE4C3C2BE7 (0.2.9.3-alpha): consensus expired at 2016-11-03T18:00:002F9A6B5ADBE91EC69F55AAFB7DC49619D31B8324 (0.2.9.3-alpha-dev): consensus expired at 2016-10-29T11:00:0021281A01279875A67C4FD2C6B125AC0587CBF9F9 (0.2.9.4-alpha): consensus expired at 2016-11-04T04:00:001685874B7B91E5C6184CF7CC5ADF8D983F150AD5 (0.2.9.2-alpha): consensus expired at 2016-09-09T13:00:001602E42D1DE3C7B3EF042F357F906DE55FA6C7C6 (0.2.9.2-alpha): consensus expired at 2016-11-02T14:00:0014F92FF956105932E9DEC5B82A7778A0B1BD9A52 (0.2.9.3-alpha): consensus expired at 2016-09-28T22:00:0010FDA609624925659622F89107C8494B6033A7FA (0.2.9.4-alpha): consensus expired at 2016-10-31T14:00:000FDAC93946A066C87BBB6A955B2893879A788CDE (0.2.9.4-alpha): consensus expired at 2016-11-04T06:00:0000AD0313351C07411F44C0A78FC54C3EBAE23265 (0.2.9.4-alpha): consensus expired at 2016-10-31T17:00:00
That's the majority of 0.2.9.4 relays. Results attached.
The fallback checks sound better. Scanning the whole network took over ten hours so not something we'd run on a daily basis. If it's sufficient to just check fallbacks that's a lot more targeted.
This script only works if your machine is in the UTC timezone.
If your machine is in a negative timezone, it succeeds when it should actually fail.
If your machine is in a positive timezone (like mine), it fails when it should actually succeed.
It needs to use datetime.datetime.utcnow(), because valid_until is in UTC.
I'm reopening this to let you weigh in on the impact, but I doubt we want to re-run the scan.
Trac: Resolution: implemented toN/A Status: closed to reopened
Gotcha, thanks for the catch. If folks want me to re-run the scan then happy to do so but think we got basically what we want from it. Feel free if you'd care to request a re-run.
Trac: Status: reopened to closed Resolution: N/Ato implemented
Can you re-run this check on microdesc consensuses with the UTC fix?
I found one relay on 0.2.9.5-alpha that has a stale microdesc consensus but good full consensus (legacy/trac#20909 (moved)). I'd like to know how prevalent this issue is.
Trac: Status: closed to reopened Resolution: implemented toN/A