Skip to content

Usability testing of Tor VPN pre-alpha

This ticket relates to the following objectives:

  • O1.4: Test wireframes and user flows with target users, identify user challenges, iterate on these designs throughout the project.

We'll conduct 6 in-person moderated usability test with 2-3 different profiles of users.

Time estimate: Large × Moderate = 6­–9 d

Results

Dates

  • Initial: May 21 to 26
  • Latest possible in Q2 (recording only): June 11-16
  • Postpone to Q3: August 6-11

References

Important bugs

Top things we want to test or learn

  • Per-app protection [BROKEN: tpo/applications/vpn#159 (closed)] (Task D)
    • Make informed decisions on which apps to route through Tor +1 +1
    • Configure only certain apps to use Tor VPN +1
    • Do users prefer to exclude or include apps? +1 +1 +1 (Tricky!)
    • Is the relationship between the super toggles (e.g. Protect all apps) and the subordinate toggles confusing? +1
  • Connection
  • Exit location [WORKS] (Task C)
    • Select an exit location +1
  • Different Tor circuit per app vs. VPN with 1 server +1 (Interview E)
    • Do people need to know about this?
    • Should we expose this higher in the UI?
  • How people perceive using Tor VPN +1 +1
    • Compared to other VPNs? (Interview F)
    • Related to the Tor brand perception (Interview G)
    • Do they look at Tor VPN without understanding its uniqueness or understand its specificities? For example, 3 hops vs. 1 hop. (Interview E and H)

User tasks

Context

Traveling to different countries

Randomized

  • A. Visit website with fake country (no censorship)
  • B. Visit website when the website is blocked (light censorship)
  • C. Send a Telegram message when Tor and Telegram are blocked (heavy censorship)
    • D. Only protect Telegram

Redacted (in Spanish): https://gitlab.torproject.org/tpo/ux/research/uploads/62918d3f4d5b83cfdd1b246107afa70b/tasks.odt

Draft interview questions

For everybody

E. How would you describe what happens to your connection to the different websites and messaging apps when using Tor VPN?

  • Can you make a drawing to explain this?
  • Are different websites and apps using the same connection or different connections? (or whatever term the person uses)
  • How would you check that in the app?

For users of VPNs

F. How would you compare Tor VPN to the other VPNs that you've used in the past?

  • What differences did you notice between Tor VPN and a regular VPN?
  • How would you compare the speed and performance compared to a regular VPNs?
  • How would you compare the privacy and security compared to a regular VPNs?
  • What do you think is better in Tor VPN, if anything?
  • What do you think is better with a regular VPN, if anything?

For users of Tor

G. What do you think about the idea of making all the Internet traffic of your mobile device go through the Tor network?

  • Does this raise any concerns or advantages in your mind?

For users of both

H. What do you think about the 3 hops used by Tor VPN, as opposed to the single hop used by a regular VPN?

  • Do you perceive this as an advantage or disadvantage, and why?

Checklist

  • Recruiting
    • Schedule slots
    • Create screening survey
    • Test screening survey
    • Send link to screening survey
    • Schedule participants using screening spreadsheet
      • Schedule 1 extra participant for no shows
    • Answer to all candidates using email templates
    • Send research information sheet to participant when confirming location
  • Preparation
    • Prepare the tasks and link from here: tasks.odt
    • Prepare testing device
    • Adapt research information sheet
    • Adapt consent form
    • Adapt checklist
      • Explicit reset procedure
      • Explicit recording procedure
    • Prepare recording setup
      • Prepare recording table for smartphone and drawings
      • Test the recording setup
      • Check available disk space
    • Prepare interview
      • Review interview script vs. research questions
      • Prepare material for drawings
    • Prepare incentives
      • Get envelopes
    • Buy snacks and drinks
    • Print
      • Research information sheet
      • Consent forms
      • Tasks
      • Interview questions
      • Testing checklist
  • Analysis
  • Wrap up
    • Share results with team
    • Share results with participants
    • Save candidates in email database
    • Add to achievements.mdwn
    • Reimburse incentives
    • Improve checklist
    • Document new networking tricks
Edited by sajolida