# robots.txt - crawler configuration file User-agent: * Disallow: /_includes/ Disallow: /_private/ Disallow: /admin/ Disallow: /css/ Disallow: /database/ Disallow: /images/ Disallow: /js/ Disallow: /login.asp Sitemap: http://www.ThaiDo.com.au/sitemap.xml