-
Type:
Improvement
-
Status: Closed (View Workflow)
-
Priority:
Minor
-
Resolution: Fixed
-
Affects Version/s: None
-
Fix Version/s: 5.0
-
Component/s: None
-
Labels:None
-
Attachments:
-
Comments:1
-
Documentation Status:Not Required
Some bots are just trouble and we keep on doing the same changes for clients over and over again.
Looking at wikipedia's list of blocked bots, I would suggest to add AT LEAST these bots:
http://en.wikipedia.org/robots.txt
Wikipedia is NOT blocking Baiduspider, but this is one that has been causing headaches for our clients as well.
- is related to
-
DS-1936 Robots.txt should include a SiteMap entry
-
- Closed
-