Skip to content

feat: Add robots.txt file

stephen requested to merge robots-txt into main

donate-neo does not currently have a robots.txt file. While it's certainly the case that web scrapers exist which will ignore such files with impunity, it is best practice to have one, and we should tee one up for those crawlers and scrapers which do respect them.

This MR adds such a file and gives it the donate-neo equivalent of what was in place on donate-paleo and what exists on the main tordonate.org site - we tell all crawlers to ignore our static assets directory.

Closes #109 (closed).

Edited by stephen

Merge request reports