Recent robots.txt change
Can someone explain what this update to robots.txt actually does?https://github.com/concrete5/concrete5/blob/master/web/robots.txt...
I know it's to help with the Google update this month and making C5 sites "mobile friendly", just wanting a bit more info :)
The change gives bots access to your themes CSS and JS directories. This way google can tell if your theme/site is mobile friendly. The old version blocked google form accessing these directories.
Here is a handy testing tool.
Test one of your sites with new and old version of robots.txt using the link above to see.