Merge Atlas and Compass into a single tool

added component::metrics/compass in Legacy / Trac owner::metrics-team in Legacy / Trac priority::low in Legacy / Trac resolution::duplicate in Legacy / Trac severity::normal in Legacy / Trac status::closed in Legacy / Trac type::enhancement in Legacy / Trac labels

I agree that Atlas and Compass have useful features that could be merged to avoid duplicating effort in the future. The features you suggest (adding columns, making them configurable, and adding an advanced search) are all doable and just need (non-trivial amounts of) developer time. I can see them to be added to either website, but I'd want to avoid them to be added to both, or we'll duplicate effort. Let me explain why this decision for extending either Atlas or Compass is hard:

Atlas focuses on presenting information about a single relay to its users. There is a search function in Atlas, but its main use is to look up a relay and present its details, including useful graphs. People could as well bookmark details pages of their favorite relays and go from there. The design is that all the hard work is either done on the client using JavaScript or on the back-end called Onionoo which does all the searching work. Atlas can only do some filtering and sorting of results it received from Onionoo, but Atlas shouldn't download the whole relay data (currently 6.6M) for each request. It would also be Onionoo that we'd have to extend to support advanced search and any other operations like sorting by the columns you mentioned.
Compass has its focus on the whole network. The motivation was to look at network diversity by grouping by country, AS, or family to see how much (exit) traffic a certain group of relays sees compared to other groups. Another goal of Compass is to track progress of deploying fast exits for sponsor J, which also requires the full network view. Compass is implemented as a server application and downloads the full relay data of 6.6M once per hour.

As I see it, we have (at least) three choices here:

Keep Compass and Atlas separate by letting Compass do the searching/filtering/sorting and Atlas displaying single relay information. Configurable columns and advanced search would go into Compass then which would continue to link to Atlas' details pages. We could even host the two applications on a single machine, so that they would appear as one service.
Extend Compass to do the advanced searching and all that, and have it also render the relay details page. Then we wouldn't need Atlas anymore.
Extend Atlas to provide advanced searching/filtering/grouping options to users and implement them in Onionoo. The we wouldn't need Compass anymore.

In the end, the decision will be made by whoever is going to do the coding work. But let's do some thinking before we decide for either design, or we might waste quite some developer time and maybe end up with a design we don't really like.

Assigning this ticket to Compass, though Atlas would be a good fit, too. But it has to go somewhere, and "- Select a component" doesn't seem right.

Trac:
Cc: N/A to hellais, gsathya
Component: - Select a component to Compass

Hi Karsten,

thanks for your comprehensive answer.

Replying to karsten:

As I see it, we have (at least) three choices here:

Keep Compass and Atlas separate by letting Compass do the searching/filtering/sorting and Atlas displaying single relay information. Configurable columns and advanced search would go into Compass then which would continue to link to Atlas' details pages. We could even host the two applications on a single machine, so that they would appear as one service.

I vote for this one :)

I would even say that simply adding a nickname/fingerprint filter option to https://compass.torproject.org/ would make the search box on https://atlas.torproject.org obsolete and you could go ahead and run atlas on the compass host and you are probably done (+ addressing other tickets regarding columns and sorting).

cypherpunk notes in legacy/trac#6704 (moved) that we could make the # column clickable to be the Atlas link rather than the fingerprint column. This makes sense once table columns are customizable, because then users might want to get rid of the fingerprint column.

Replying to cypherpunks:

I would even say that simply adding a nickname/fingerprint filter option to https://compass.torproject.org/ would make the search box on https://atlas.torproject.org obsolete and you could go ahead and run atlas on the compass host and you are probably done (+ addressing other tickets regarding columns and sorting).

Not quite. Atlas also has a very nice feature to filter relays using the fields at the bottom of the table.

Reducing priority to minor, because this is really nice to have, but it's also a potentially big task and I think we should work on the other tickets first.

Trac:
Priority: normal to minor

I have been thinking about this. I think a nicer solution would be to make Compass just spit out JSON files and not HTML. Then extend Atlas to use this as a backend. Atlas should be able to talk to both Onionoo(for specific relay info) and Compass(for processed data). Essentially Compass becomes an extension to Onionoo. We might later want to extend Onionoo to do what Compass does if we don't want to maintain two separate services.

This is a pretty big task and I'm not sure what the plans for Atlas are. I remember hellais saying something about backbone.js being not good enough and it's now hard to extend Atlas. Is there going to be some Atlas rewrite or something?

Trac:
Status: new to needs_information

Replying to gsathya:

I have been thinking about this. I think a nicer solution would be to make Compass just spit out JSON files and not HTML. Then extend Atlas to use this as a backend. Atlas should be able to talk to both Onionoo(for specific relay info) and Compass(for processed data). Essentially Compass becomes an extension to Onionoo. We might later want to extend Onionoo to do what Compass does if we don't want to maintain two separate services.

I like the idea of moving the Compass functionality to the back-end. But two back-end services with Onionoo and Compass doesn't make much sense. Everything that Compass knows is already known to Onionoo. This functionality should then be added to Onionoo.

Can you describe the interface that you have in mind? What queries would the extended Atlas send to the extended Onionoo? Maybe I can help integrating those queries into Onionoo's current interface.

This is a pretty big task and I'm not sure what the plans for Atlas are. I remember hellais saying something about backbone.js being not good enough and it's now hard to extend Atlas. Is there going to be some Atlas rewrite or something?

I don't know of any plans to rewrite Atlas, and I don't see how we have the developer resources to do that. Can you be more specific why it's hard to extend Atlas to support Compass' front-end functionality, assuming that Onionoo would support Compass' back-end functionality?

I agree that this is a pretty big task. But that shouldn't us from making plans.

Replying to karsten:

I like the idea of moving the Compass functionality to the back-end. But two back-end services with Onionoo and Compass doesn't make much sense. Everything that Compass knows is already known to Onionoo. This functionality should then be added to Onionoo.

But Compass and Onionoo do different things. I don't know if any other application is going to use the data that Compass requires. I guess maintaining two different backend services are going to be a pain, so yes we should probably merge the two.

Can you describe the interface that you have in mind? What queries would the extended Atlas send to the extended Onionoo? Maybe I can help integrating those queries into Onionoo's current interface.

I think pretty much what compass.py does. I will think more about a more optimized interface such that we can offload some kind of processing work to the client side js.

I don't know of any plans to rewrite Atlas, and I don't see how we have the developer resources to do that. Can you be more specific why it's hard to extend Atlas to support Compass' front-end functionality, assuming that Onionoo would support Compass' back-end functionality?

I haven't really looked at the Atlas code, but I don't suspect it's going to be impossible to do this. We should also figure out how to make this intuitive for people to use so that they don't get confused.

I agree that this is a pretty big task. But that shouldn't us from making plans.

+1

Replying to gsathya:

Replying to karsten:

I like the idea of moving the Compass functionality to the back-end. But two back-end services with Onionoo and Compass doesn't make much sense. Everything that Compass knows is already known to Onionoo. This functionality should then be added to Onionoo.

But Compass and Onionoo do different things. I don't know if any other application is going to use the data that Compass requires. I guess maintaining two different backend services are going to be a pain, so yes we should probably merge the two.

Can you describe the interface that you have in mind? What queries would the extended Atlas send to the extended Onionoo? Maybe I can help integrating those queries into Onionoo's current interface.

I think pretty much what compass.py does. I will think more about a more optimized interface such that we can offload some kind of processing work to the client side js.

This doesn't solve the discussion, but I should mention that my rewrite of compass.py already makes it spit out JSON rather than text strings, so that might be a little bit less work if you wanted to turn it into a service. When you run it from the command line it looks like what compass.py used to do, but that's just because of the print_selection call at the end. For example, loading compass.torproject.org/result.json?exit_filter=all_relays&links&sort=cw&sort_reverse&country=&ases=AS3320 directly will just give you the JSON.

The client side JS is currently handling all of the display and input, but isn't actually doing any processing.

I don't know of any plans to rewrite Atlas, and I don't see how we have the developer resources to do that. Can you be more specific why it's hard to extend Atlas to support Compass' front-end functionality, assuming that Onionoo would support Compass' back-end functionality?

I haven't really looked at the Atlas code, but I don't suspect it's going to be impossible to do this. We should also figure out how to make this intuitive for people to use so that they don't get confused.

I agree that this is a pretty big task. But that shouldn't us from making plans.

+1

We have discussed this a bit with gsathya, karsten and mo.

I think we have reached a consensus on the fact that compass and atlas should be merged into one tool in the following way.

Compass + Atlas (Adventurer?) should be only a frontend to onionoo.

What is currently missing inside of Atlas that is part of Compass is the ability to perform advanced searches for relays. This would be implemented by making the search functionality support more advanced parameter searches.

The parameter based filtering logic that is currently part of compass should end up inside of onionoo.

The currently missing features in onionoo are:

"select only relays suitable for exit position", this should probably implement a search parameter based on the value of "flag".
"select only relays suitable for guard position", this should also be implemented a parameter on the value of "flag"
"select relays by AS number"

I would suggest we port Atlas to use angular.js and integrate the parts of relevant code from Compass.

Adventurer should only be a front end written in HTML/JS and not do any of the heavy lifting of parsing the onionoo responses and extracting the results (in python). All the necessary filtering and parsing activity should be done in the browser.

Trac:
Cc: hellais, gsathya to hellais, gsathya, mo

Onionoo now supports an "as" and a "flag" parameter to support the missing features. See the protocol page for details. Let me know if something else is missing.

Non-technical remark: maybe we should call the new tool either Atlas or Compass, rather than finding a new name for it. I'd guess that Atlas is better known to people than Compass, so maybe we should call the new thing Atlas and add a redirect from compass.tpo to atlas.tpo. Just a thought.

Tweak summary a bit.

Trac:
Reviewer: N/A to N/A
Summary: merge atlas and compass? to Merge Atlas and Compass into a single tool
Sponsor: N/A to N/A
Severity: N/A to Normal