flâneur

Can domains be used in robots.txt? - Stack Overflow

stackoverflow.com · 1,254 words · saved by 1 readers

Ask questions, find answers and collaborate at work with Stack Overflow for Teams. Explore Teams Find centralized, trusted content and collaborate around the technologies you use most. Teams Q&A for work Connect and share knowledge within a single location that is structured and easy to search. Get early access and see previews of new features. We have a development server at dev.example.com that is being indexed by Google. We are using AWS Lightsail to duplicate the development server to our production environment in totality — the same robots.txt file is used on both dev.example.com and example.com. Google's robots.txt documentation doesn't explicitly state whether root domains can be defined. Can I implement domain specific rules to the robots.txt file? For example, is this acceptable: To add, this can be resolved through .htaccess rewrite engine — my question is specifically about robots.txt. No, you can't specify domain in robots.txt. Disallow: https://dev.example.com/

Can domains be used in robots.txt? - Stack Overflow New: Stack Overflow For Agents. The next generation of knowledge exchange. Learn more Collectives™ on Stack Overflow Find centralized, trusted content and collaborate around the technologies you use most. Learn more about Collectives Stack Internal Knowledge at work Bring the best of human thought and AI automation together at your work. Explore Stack Internal Can domains be used in robots.txt? Ask Question Asked 4 years, 3 months ago Modified 2 years, 11 months ago Viewed 6k times 5 We have a development server at dev.example.com that is bei

saved by

related reading