Robots.txt and my Core Concrete5 is getting indexed by Google!!!
The problem is - it doesn't look like the robots.txt has the /updates folder as disallow.
Just wanted to bring that to people's attention. I assume I'm not the only one that updated this way and will soon be having my core files indexed on Google! similar to the thread below:
Besides adding the /updates folder to the robots.txt - does anyone have any suggestions, or should this suffice? I also think I need to turn off directory listing.
But even if google indexes the updates directory, it's not going to cause a lot of problems, is it? Don't get me wrong, I'm not saying it's a good thing that robots.txt doesn't contain this directory, just saying that it's not the end of the world..
Add it to robots.txt and you're fine..
I'd actually say it's a lot more important that directory browsing is disabled!
Probably wouldn't be a bad idea to stick this into the default robots.txt going foward...