Your Drupal robots.txt Changes Aren't Showing Up: Here's Why
You've installed the robotstxt module, made your changes in the Drupal admin at /admin/config/search/robotstxt, cleared the cache, run cron — and the live robots.txt file is still showing old content. Before you go down a rabbit hole, here are the most common culprits.
1. Composer Scaffold Is Regenerating the File on Every Deploy
If your project uses robots.txt (which most Composer-based Drupal projects do), robots.txt is one of the scaffold files it copies into your webroot on every composer install. This means even if you delete it manually, it comes back on the next deploy, silently overriding the robotstxt module every time.
The fix is to tell the scaffold plugin to skip robots.txt entirely. In your composer.json, add a false entry for the file under file-mapping:
"drupal-scaffold": { "file-mapping": { "[web-root]/robots.txt": false }, "locations": { "web-root": "web/" }}
If your project already has a file-mapping section, just add the robots.txt...
Copyright of this story solely belongs to hackernoon.com. To see the full text click HERE
