Robots.txt to Become an Official Standard After 25 Years
Last updated on July 2, 2019 at 01:46 pm In a series of tweets by Google Webmasters, they have announced their proposal of a draft stating that Robots.txt is well on its way from becoming a de facto standard to an official one. To quote Martijn, “This is especially handy if you have large archives,…