AdsBot-Google does not respect robots.txt
I noticed something strange today. AdsBot does not respect robots.txt as you would expect. AdsBot ignores general rules set in the robots.txt file. So for example, if your robots.txt file looks something like this: You would expect the bot not to crawl any pages in the /javascript/ folder. However, AdsBot does crawl those pages. If you specifically name AdsBot in […]
