You can see what a bot sees for a thought by adding ?bot=1 to the end of a thought url.

Bots are detected in config.php and the php function isBot() identifies bots according to a regular expression which is manually updated periodically as new bots are discovered.

Detected bots are recorded in the bots table with allow=1 for bots that were allowed through and 0 for ones that were rejected.

Allowed bots are served the content of td/php/botthought.php  on thoughts. Bots are only allowed to access thoughts.

The sitemap.txt file is updated with the last 2000 thoughts whenever a new thought is created. This file is referenced in robots.txt so that all bots can find it to index new thoughts from. Bots are also allowed to find thoughts from the home page tag cloud. Note that to be included in the sitemap a thought must have statuspublish” (not private or muffle) and not be a draft or in the back room.

Bots attempting to follow links to anything other than a thought will receive an instant 403 Forbidden error with very little code having been run and only the bot tracking database query. Very efficient.

Currently widget content does not appear to bots, only thought body and comment body content.