Google now respond to ‘Unavailable’ meta tag
After an announcement a couple of weeks ago Google will now recoginise and act on the new ‘Unavailable’ Meta Tag.
By adding in the meta tag information into each page you are telling Google not to crawl the page and to remove it from their index after that date.
Google estimate that it will take a day for the page to be removed.
So in the example below you are telling Google to remove the page after 3pm Eastern Standard Time on the 30th August.
<META NAME="GOOGLEBOT" CONTENT="unavailable_after: 30-Aug-2007 15:00:00 EST">
The date and time is specified in the RFC 850 format and we will check to see whether ‘GMT’ is supported but we would expect it to be so. Even so, you can still do the maths and take into account the 5 hours differance!
Currently this only applies to web pages.
The application of this could be on ‘special offer’ promotion pages or products which have a known expiry date.
There are also a new X-Robots-Tags for prohibiting indexing of non HTML documents such as PDF files, audio, xls but that is another story which we will have more on later!