fix: Serve robots.txt from Django's root, not project root
!152 (merged) was a well-intentioned implementation of robots.txt, but as was kindly pointed out in that thread, Django needs to route traffic from a given URL to a specific file, and that MR just put the file in the project root. Not quite the same thing! This MR adjusts this implementation in two ways:
-
robots.txt
has been moved intotorproject.templates
and converted to a Jinja template - we don't take advantage of this now, but this future-proofs this file (in case, for example,donate-neo
winds up needing to combat a variety of bad bot traffic sources which might be easier kept as data and iterated over, rather than duplicating it in plaintext). -
urls.py
has been updated to provide routing from/robots.txt
to our template, and told to serve it as plaintext.
Testing this locally gives us the expected output, and response headers confirm this content is being served as Content-Type: text/plain
. Additionally, Jinja template tags can be mixed in with the plaintext in this file, and the resultant Jinja-rendered output will still be served as plaintext.
Follows up on !152 (merged).