Skip to content
Snippets Groups Projects
New look: Off

Onionprobe: initial improvements

    • Closed Issue created by Silvio Rhatto @rhatto
    • Try Onionscan for inspiration/ideas.
    • Refine Onionprobe metrics collection.
    • Implement other needed refinements for putting it in production to monitor Sponsor 123 sites.
    ✓ 3 of 3 checklist items completed · Edited

    Designs

      Child items ...

      2. Activity

      • Silvio Rhatto assigned to @rhatto

        assigned to @rhatto

      • Silvio Rhatto moved from onion-support#58 (moved)

        moved from onion-support#58 (moved)

      • Silvio Rhatto changed due date to April 01, 2022

        changed due date to April 01, 2022

      • Silvio Rhatto added Next Onion Services labels

        added Next Onion Services labels

      • Silvio Rhatto assigned to @rhatto

        assigned to @rhatto

      • Silvio Rhatto marked this issue as related to onion-support#50 (moved)

        marked this issue as related to onion-support#50 (moved)

      • Silvio Rhatto mentioned in issue onion-support#50 (moved)

        mentioned in issue onion-support#50 (moved)

      • Silvio Rhatto changed title from Onionprobe: improvements to Onionprobe improvements

        changed title from Onionprobe: improvements to Onionprobe improvements

      • Silvio Rhatto
        Silvio Rhatto @rhatto ·
        Author Owner

        Task estimation

        • Complexity: medium (1 day)
        • Uncertainty: low (x1.1)
        • Reference
      • Silvio Rhatto changed time estimate to 9h

        changed time estimate to 9h

      • Silvio Rhatto changed the description

        changed the description

      • Silvio Rhatto added Doing label and removed Next label

        added Doing label and removed Next label

      • Silvio Rhatto
        Silvio Rhatto @rhatto ·
        Author Owner

        Onionscan test

        About

        Using this fork due to these issues:

        Installation

        $ go get github.com/CypherpunkSamurai/onionscan

        Tests

        Test using the official Tor website .onion:

        $ onionscan http://2gzyxa5ihm7nsggfxnu52rck2vv4rvmdlkiu3zzui5du4xyclen53wid.onion
2022/03/24 19:56:55 Creating Database Bucket crawls...
2022/03/24 19:56:55 Indexing URL in crawls...
2022/03/24 19:56:55 Creating Database Bucket relationships...
2022/03/24 19:56:56 Indexing Identifier in relationships...
2022/03/24 19:56:56 Indexing Onion in relationships...
2022/03/24 19:56:56 Indexing Type in relationships...
2022/03/24 19:56:56 Indexing From in relationships...
2022/03/24 19:56:56 Database Setup Complete
2022/03/24 19:56:56 Starting Scan of http://2gzyxa5ihm7nsggfxnu52rck2vv4rvmdlkiu3zzui5du4xyclen53wid.onion
2022/03/24 19:56:56 This might take a few minutes..

2022/03/24 19:59:38 ERROR: Document is too large. Max: `2097152`, Given: `4191658`
2022/03/24 19:59:47 ERROR: Document is too large. Max: `2097152`, Given: `4191658`
2022/03/24 19:59:56 ERROR: Document is too large. Max: `2097152`, Given: `3985668`
2022/03/24 20:00:03 ERROR: Document is too large. Max: `2097152`, Given: `3985666`
2022/03/24 20:00:10 ERROR: Document is too large. Max: `2097152`, Given: `3987692`
2022/03/24 20:00:14 ERROR: Document is too large. Max: `2097152`, Given: `3987692`
[...]

        Test using the Snowflake .onion:

        $ onionscan http://oljlphash3bpqtrvqpr5gwzrhroziw4mddidi5d2qa4qjejcbrmoypqd.onion
2022/03/24 20:05:38 Starting Scan of http://oljlphash3bpqtrvqpr5gwzrhroziw4mddidi5d2qa4qjejcbrmoypqd.onion
2022/03/24 20:05:38 This might take a few minutes..

2022/03/24 20:06:15 ERROR: Document is too large. Max: `2097152`, Given: `2999322`
2022/03/24 20:06:41 ERROR: Document is too large. Max: `2097152`, Given: `2999322`
2022/03/24 20:06:44 ERROR: Document is too large. Max: `2097152`, Given: `2999322`
2022/03/24 20:06:46 ERROR: Document is too large. Max: `2097152`, Given: `2999322`
2022/03/24 20:06:50 ERROR: Document is too large. Max: `2097152`, Given: `2999322`
2022/03/24 20:07:05 Inserting oljlphash3bpqtrvqpr5gwzrhroziw4mddidi5d2qa4qjejcbrmoypqd.onion --- clearnetlink ---> https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/wikis/home (uri)
2022/03/24 20:07:05 Inserting oljlphash3bpqtrvqpr5gwzrhroziw4mddidi5d2qa4qjejcbrmoypqd.onion --- clearnetlink ---> https://www.torproject.org/download/ (uri)
2022/03/24 20:07:05 Inserting oljlphash3bpqtrvqpr5gwzrhroziw4mddidi5d2qa4qjejcbrmoypqd.onion --- clearnetlink ---> https://chrome.google.com/webstore/detail/snowflake/mafpmfcccpbjnhfhjnllmmalhifmlcie (uri)
2022/03/24 20:07:05 Inserting oljlphash3bpqtrvqpr5gwzrhroziw4mddidi5d2qa4qjejcbrmoypqd.onion --- clearnetlink ---> https://community.torproject.org/relay/setup/snowflake/standalone/ (uri)
2022/03/24 20:07:05 Inserting oljlphash3bpqtrvqpr5gwzrhroziw4mddidi5d2qa4qjejcbrmoypqd.onion --- clearnetlink ---> https://gitlab.onionize.space/ (uri)
2022/03/24 20:07:05 Inserting oljlphash3bpqtrvqpr5gwzrhroziw4mddidi5d2qa4qjejcbrmoypqd.onion --- clearnetlink ---> https://anonticket.onionize.space/ (uri)
2022/03/24 20:07:05 Inserting oljlphash3bpqtrvqpr5gwzrhroziw4mddidi5d2qa4qjejcbrmoypqd.onion --- clearnetlink ---> https://support.torproject.org/censorship/ (uri)
2022/03/24 20:07:05 Inserting oljlphash3bpqtrvqpr5gwzrhroziw4mddidi5d2qa4qjejcbrmoypqd.onion --- clearnetlink ---> https://www.torproject.org/ (uri)
2022/03/24 20:07:05 Inserting oljlphash3bpqtrvqpr5gwzrhroziw4mddidi5d2qa4qjejcbrmoypqd.onion --- clearnetlink ---> https://support.torproject.org/ (uri)
2022/03/24 20:07:05 Inserting oljlphash3bpqtrvqpr5gwzrhroziw4mddidi5d2qa4qjejcbrmoypqd.onion --- clearnetlink ---> https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues (uri)
2022/03/24 20:07:05 Inserting oljlphash3bpqtrvqpr5gwzrhroziw4mddidi5d2qa4qjejcbrmoypqd.onion --- clearnetlink ---> https://addons.mozilla.org/en-US/firefox/addon/torproject-snowflake/ (uri)
2022/03/24 20:07:05 Inserting oljlphash3bpqtrvqpr5gwzrhroziw4mddidi5d2qa4qjejcbrmoypqd.onion --- links ---> qa4qjejcbrmoypqd.onion (uri)
2022/03/24 20:07:05 Inserting oljlphash3bpqtrvqpr5gwzrhroziw4mddidi5d2qa4qjejcbrmoypqd.onion --- crawl ---> 3252044374202655258 (database-id)
2022/03/24 20:07:05 Inserting oljlphash3bpqtrvqpr5gwzrhroziw4mddidi5d2qa4qjejcbrmoypqd.onion --- crawl ---> 2566886044149954282 (database-id)
2022/03/24 20:07:05 Inserting oljlphash3bpqtrvqpr5gwzrhroziw4mddidi5d2qa4qjejcbrmoypqd.onion --- crawl ---> 975087538070709844 (database-id)
2022/03/24 20:07:05 Inserting oljlphash3bpqtrvqpr5gwzrhroziw4mddidi5d2qa4qjejcbrmoypqd.onion --- crawl ---> 8611190104506858967 (database-id)
2022/03/24 20:07:05 Inserting oljlphash3bpqtrvqpr5gwzrhroziw4mddidi5d2qa4qjejcbrmoypqd.onion --- crawl ---> 2244119072103791535 (database-id)
2022/03/24 20:07:05 Inserting oljlphash3bpqtrvqpr5gwzrhroziw4mddidi5d2qa4qjejcbrmoypqd.onion --- crawl ---> 6205816337361538989 (database-id)
2022/03/24 20:07:05 Inserting oljlphash3bpqtrvqpr5gwzrhroziw4mddidi5d2qa4qjejcbrmoypqd.onion --- crawl ---> 6327994479490919222 (database-id)
2022/03/24 20:07:05 Inserting oljlphash3bpqtrvqpr5gwzrhroziw4mddidi5d2qa4qjejcbrmoypqd.onion --- crawl ---> 9183229070864729916 (database-id)
2022/03/24 20:07:05 Inserting oljlphash3bpqtrvqpr5gwzrhroziw4mddidi5d2qa4qjejcbrmoypqd.onion --- crawl ---> 9155734715090302355 (database-id)
2022/03/24 20:07:05 Inserting oljlphash3bpqtrvqpr5gwzrhroziw4mddidi5d2qa4qjejcbrmoypqd.onion --- crawl ---> 7771804545677986349 (database-id)
2022/03/24 20:07:05 Inserting oljlphash3bpqtrvqpr5gwzrhroziw4mddidi5d2qa4qjejcbrmoypqd.onion --- crawl ---> 1441234593527817871 (database-id)
2022/03/24 20:07:05 Inserting oljlphash3bpqtrvqpr5gwzrhroziw4mddidi5d2qa4qjejcbrmoypqd.onion --- crawl ---> 1631310578631456192 (database-id)
2022/03/24 20:07:05 Inserting oljlphash3bpqtrvqpr5gwzrhroziw4mddidi5d2qa4qjejcbrmoypqd.onion --- crawl ---> 4542044602408668221 (database-id)
2022/03/24 20:07:05 Inserting oljlphash3bpqtrvqpr5gwzrhroziw4mddidi5d2qa4qjejcbrmoypqd.onion --- crawl ---> 2869434301312096496 (database-id)
2022/03/24 20:07:05 Inserting oljlphash3bpqtrvqpr5gwzrhroziw4mddidi5d2qa4qjejcbrmoypqd.onion --- crawl ---> 268265378923406671 (database-id)
2022/03/24 20:07:05 Inserting oljlphash3bpqtrvqpr5gwzrhroziw4mddidi5d2qa4qjejcbrmoypqd.onion --- crawl ---> 2944970960098863230 (database-id)
2022/03/24 20:07:05 Inserting oljlphash3bpqtrvqpr5gwzrhroziw4mddidi5d2qa4qjejcbrmoypqd.onion --- crawl ---> 3030545405875762529 (database-id)
2022/03/24 20:07:05 Inserting oljlphash3bpqtrvqpr5gwzrhroziw4mddidi5d2qa4qjejcbrmoypqd.onion --- crawl ---> 1248148679358504568 (database-id)
2022/03/24 20:07:05 Inserting oljlphash3bpqtrvqpr5gwzrhroziw4mddidi5d2qa4qjejcbrmoypqd.onion --- crawl ---> 4856931331095718084 (database-id)
2022/03/24 20:07:05 Inserting oljlphash3bpqtrvqpr5gwzrhroziw4mddidi5d2qa4qjejcbrmoypqd.onion --- crawl ---> 1736585779089822282 (database-id)
2022/03/24 20:07:05 Inserting oljlphash3bpqtrvqpr5gwzrhroziw4mddidi5d2qa4qjejcbrmoypqd.onion --- crawl ---> 9030851906836485328 (database-id)
2022/03/24 20:07:05 Inserting oljlphash3bpqtrvqpr5gwzrhroziw4mddidi5d2qa4qjejcbrmoypqd.onion --- crawl ---> 3595159136557828177 (database-id)
2022/03/24 20:07:05 Inserting oljlphash3bpqtrvqpr5gwzrhroziw4mddidi5d2qa4qjejcbrmoypqd.onion --- crawl ---> 8117486423974863734 (database-id)
2022/03/24 20:07:05 Inserting oljlphash3bpqtrvqpr5gwzrhroziw4mddidi5d2qa4qjejcbrmoypqd.onion --- crawl ---> 7592397972693726449 (database-id)
2022/03/24 20:07:05 Inserting oljlphash3bpqtrvqpr5gwzrhroziw4mddidi5d2qa4qjejcbrmoypqd.onion --- crawl ---> X-Frame-Options:sameorigin (http-header)
2022/03/24 20:07:05 Inserting oljlphash3bpqtrvqpr5gwzrhroziw4mddidi5d2qa4qjejcbrmoypqd.onion --- crawl ---> Accept-Ranges:bytes (http-header)
2022/03/24 20:07:05 Inserting oljlphash3bpqtrvqpr5gwzrhroziw4mddidi5d2qa4qjejcbrmoypqd.onion --- crawl ---> Date:Thu, 24 Mar 2022 20:06:10 GMT (http-header)
2022/03/24 20:07:05 Inserting oljlphash3bpqtrvqpr5gwzrhroziw4mddidi5d2qa4qjejcbrmoypqd.onion --- crawl ---> Etag:"15a0-5c716180a564d-gzip" (http-header)
2022/03/24 20:07:05 Inserting oljlphash3bpqtrvqpr5gwzrhroziw4mddidi5d2qa4qjejcbrmoypqd.onion --- crawl ---> Last-Modified:Wed, 14 Jul 2021 14:23:10 GMT (http-header)
2022/03/24 20:07:05 Inserting oljlphash3bpqtrvqpr5gwzrhroziw4mddidi5d2qa4qjejcbrmoypqd.onion --- crawl ---> Referrer-Policy:no-referrer (http-header)
2022/03/24 20:07:05 Inserting oljlphash3bpqtrvqpr5gwzrhroziw4mddidi5d2qa4qjejcbrmoypqd.onion --- crawl ---> Server:Apache (http-header)
2022/03/24 20:07:05 Inserting oljlphash3bpqtrvqpr5gwzrhroziw4mddidi5d2qa4qjejcbrmoypqd.onion --- crawl ---> X-Content-Type-Options:nosniff (http-header)
2022/03/24 20:07:05 Inserting oljlphash3bpqtrvqpr5gwzrhroziw4mddidi5d2qa4qjejcbrmoypqd.onion --- crawl ---> Content-Type:text/html (http-header)
2022/03/24 20:07:05 Inserting oljlphash3bpqtrvqpr5gwzrhroziw4mddidi5d2qa4qjejcbrmoypqd.onion --- crawl ---> Vary:Accept-Encoding (http-header)
2022/03/24 20:07:05 Inserting oljlphash3bpqtrvqpr5gwzrhroziw4mddidi5d2qa4qjejcbrmoypqd.onion --- crawl ---> X-Xss-Protection:1 (http-header)
2022/03/24 20:07:05 Inserting oljlphash3bpqtrvqpr5gwzrhroziw4mddidi5d2qa4qjejcbrmoypqd.onion --- crawl ---> Snowflake (page-info)
2022/03/24 20:07:05 Inserting oljlphash3bpqtrvqpr5gwzrhroziw4mddidi5d2qa4qjejcbrmoypqd.onion --- crawl ---> 8223134970508696012 (database-id)
2022/03/24 20:07:05 Inserting oljlphash3bpqtrvqpr5gwzrhroziw4mddidi5d2qa4qjejcbrmoypqd.onion --- crawl ---> 3737814334652899672 (database-id)
2022/03/24 20:07:05 Inserting oljlphash3bpqtrvqpr5gwzrhroziw4mddidi5d2qa4qjejcbrmoypqd.onion --- crawl ---> 3944036900546750893 (database-id)
2022/03/24 20:07:05 Inserting oljlphash3bpqtrvqpr5gwzrhroziw4mddidi5d2qa4qjejcbrmoypqd.onion --- crawl ---> 4967927802199449175 (database-id)
2022/03/24 20:07:05 Inserting oljlphash3bpqtrvqpr5gwzrhroziw4mddidi5d2qa4qjejcbrmoypqd.onion --- crawl ---> 6842298708899640630 (database-id)
2022/03/24 20:07:05 Inserting oljlphash3bpqtrvqpr5gwzrhroziw4mddidi5d2qa4qjejcbrmoypqd.onion --- crawl ---> 8038567456247050492 (database-id)
2022/03/24 20:07:05 Inserting oljlphash3bpqtrvqpr5gwzrhroziw4mddidi5d2qa4qjejcbrmoypqd.onion --- crawl ---> 5657258228939218717 (database-id)
2022/03/24 20:07:05 Inserting oljlphash3bpqtrvqpr5gwzrhroziw4mddidi5d2qa4qjejcbrmoypqd.onion --- crawl ---> 7703834967621295635 (database-id)
2022/03/24 20:07:05 Inserting oljlphash3bpqtrvqpr5gwzrhroziw4mddidi5d2qa4qjejcbrmoypqd.onion --- crawl ---> 8669697040848513484 (database-id)
2022/03/24 20:07:05 Inserting oljlphash3bpqtrvqpr5gwzrhroziw4mddidi5d2qa4qjejcbrmoypqd.onion --- crawl ---> 8027530282210985657 (database-id)
2022/03/24 20:07:05 Inserting oljlphash3bpqtrvqpr5gwzrhroziw4mddidi5d2qa4qjejcbrmoypqd.onion --- crawl ---> 5612498908298851322 (database-id)
2022/03/24 20:07:05 Inserting oljlphash3bpqtrvqpr5gwzrhroziw4mddidi5d2qa4qjejcbrmoypqd.onion --- crawl ---> 3628705917192203987 (database-id)
2022/03/24 20:07:05 Inserting oljlphash3bpqtrvqpr5gwzrhroziw4mddidi5d2qa4qjejcbrmoypqd.onion --- crawl ---> 5883191873186006046 (database-id)
2022/03/24 20:07:05 Inserting oljlphash3bpqtrvqpr5gwzrhroziw4mddidi5d2qa4qjejcbrmoypqd.onion --- crawl ---> 2534364399664997097 (database-id)
2022/03/24 20:07:05 Inserting oljlphash3bpqtrvqpr5gwzrhroziw4mddidi5d2qa4qjejcbrmoypqd.onion --- crawl ---> 6242787138914129377 (database-id)
2022/03/24 20:07:05 Inserting oljlphash3bpqtrvqpr5gwzrhroziw4mddidi5d2qa4qjejcbrmoypqd.onion --- crawl ---> 3254479770255787618 (database-id)
2022/03/24 20:07:05 Inserting oljlphash3bpqtrvqpr5gwzrhroziw4mddidi5d2qa4qjejcbrmoypqd.onion --- snapshot ---> tor-logo@2x.png (email-address)
2022/03/24 20:07:05 Updating oljlphash3bpqtrvqpr5gwzrhroziw4mddidi5d2qa4qjejcbrmoypqd.onion --- snapshot ---> tor-logo@2x.png (email-address)
2022/03/24 20:07:05 Updating oljlphash3bpqtrvqpr5gwzrhroziw4mddidi5d2qa4qjejcbrmoypqd.onion --- snapshot ---> tor-logo@2x.png (email-address)
2022/03/24 20:07:05 Updating oljlphash3bpqtrvqpr5gwzrhroziw4mddidi5d2qa4qjejcbrmoypqd.onion --- snapshot ---> tor-logo@2x.png (email-address)
2022/03/24 20:07:05 Updating oljlphash3bpqtrvqpr5gwzrhroziw4mddidi5d2qa4qjejcbrmoypqd.onion --- snapshot ---> tor-logo@2x.png (email-address)
--------------- OnionScan Report ---------------
Generating Report for: oljlphash3bpqtrvqpr5gwzrhroziw4mddidi5d2qa4qjejcbrmoypqd.onion

Info: Found Identities
         Items Identified:

         tor-logo@2x.png

Low Risk: Small number of open directories were discovered!
         Why this is bad: Open directories can reveal the existence of files not linked from the sites source code. Most of the time this is benign, but sometimes operators forget to clean up more sensitive folders.
         To fix, use .htaccess rules or equivalent to make reading directories listings forbidden. Quick Fix (Disable indexing globally) for Debian / Ubuntu running Apache: a2dismod autoindex as root.
         Items Identified:

         /assets

        Testing a valid .onion address but whose descriptor is not currently published:

        $ onionscan http://4vpedii7ikef73dteo7xiy7oudi4ebp5iqqryd66powkwqxtp5ifwqqd.onion
2022/03/24 20:14:21 Starting Scan of http://4vpedii7ikef73dteo7xiy7oudi4ebp5iqqryd66powkwqxtp5ifwqqd.onion
2022/03/24 20:14:21 This might take a few minutes..

--------------- OnionScan Report ---------------
Generating Report for: 4vpedii7ikef73dteo7xiy7oudi4ebp5iqqryd66powkwqxtp5ifwqqd.onion

No risks were found.

        Testing with a probably invalid and non-existing/non-published .onion address:

        $ onionscan http://4vpedii7ikef73dteo7xiy7oudi4ebp5iqqryd66powkwqxtp5ifwaaa.onion
2022/03/24 20:21:45 Starting Scan of http://4vpedii7ikef73dteo7xiy7oudi4ebp5iqqryd66powkwqxtp5ifwaaa.onion
2022/03/24 20:21:45 This might take a few minutes..

--------------- OnionScan Report ---------------
Generating Report for: 4vpedii7ikef73dteo7xiy7oudi4ebp5iqqryd66powkwqxtp5ifwaaa.onion

No risks were found.

        Conclusions

        • Onionscan is currently unmaintained.
        • Use cases are different:
          • Onionscan aims at finding common pitfalls and issues in configured Onion Services.
          • Onionprobe is currently focused in testing Onion Services reachability.

        Future changes could be implemented at Onionprobe to make additional tests and pointing configuration issues in Onion Services, but that's not the goal of the current iteration.

        Edited by Silvio Rhatto
      • Silvio Rhatto marked the checklist item Try Onionscan for inspiration/ideas. as completed

        marked the checklist item Try Onionscan for inspiration/ideas. as completed

      • Silvio Rhatto changed the description

        changed the description

      • Silvio Rhatto added 4h 48m of time spent

        added 4h 48m of time spent

      • Silvio Rhatto marked the checklist item Refine Onionprobe metrics collection. as completed

        marked the checklist item Refine Onionprobe metrics collection. as completed

      • Silvio Rhatto added 8h 44m of time spent

        added 8h 44m of time spent

      • Loading
      • Loading
      • Loading
      • Loading
      • Loading
      • Loading
      • Loading
      • Loading
      • Loading
      • Loading
      Please register or sign in to reply