Skip to content

fix: Serve robots.txt from Django's root, not project root

stephen requested to merge robots-txt-but-reachable into main

!152 (merged) was a well-intentioned implementation of robots.txt, but as was kindly pointed out in that thread, Django needs to route traffic from a given URL to a specific file, and that MR just put the file in the project root. Not quite the same thing! This MR adjusts this implementation in two ways:

  • robots.txt has been moved into torproject.templates and converted to a Jinja template - we don't take advantage of this now, but this future-proofs this file (in case, for example, donate-neo winds up needing to combat a variety of bad bot traffic sources which might be easier kept as data and iterated over, rather than duplicating it in plaintext).
  • urls.py has been updated to provide routing from /robots.txt to our template, and told to serve it as plaintext.

Testing this locally gives us the expected output, and response headers confirm this content is being served as Content-Type: text/plain. Additionally, Jinja template tags can be mixed in with the plaintext in this file, and the resultant Jinja-rendered output will still be served as plaintext.

Follows up on !152 (merged).

Merge request reports

Loading