# file: robots.txt , V1.0 2007/05/06 Created by YiGe&Wispom # www.gkcity.com # 按照robots.txt的标准写法,规定一些不允许爬虫爬的页面或目录。 # robots.txt 的写法参照 # Format is: # User-agent: # Disallow: | Baiduspider|iaskspider|Yahoo! |Googlebot|Msnbot|Alexibot|Sogou\+web\+spider|ia_archiver|sohu-search|soso|163 # ----------------------------------------------------------------------------- User-agent: * Crawl-delay: 100 Disallow: /bin/ Disallow: /App_Themes/ Disallow: /App_Data/ Disallow: /Code/ Disallow: /Security/ Disallow: /CSS/ Disallow: /Member/ Disallow: /JavaScript/ Disallow: /Services/ Disallow: /ConfigFiles/ Disallow: /UserControls/ Disallow: /Default.Master Disallow: /ShoppingCart.htm Disallow: /InquiryCart.htm Disallow: /ShoppingCartSignin.htm Disallow: /InquiryCart.aspx Disallow: /ShoppingCart.aspx Disallow: /ShoppingCartSignin.aspx Disallow: /IM.aspx Disallow: /ViewImg.aspx Disallow: /ScriptResource.axd Disallow: /WebResource.axd Sitemap: http://www.gkcity.com/sitemap1.xml.gz Sitemap: http://www.gkcity.com/sitemap2.xml.gz Sitemap: http://www.gkcity.com/sitemap3.xml.gz Sitemap: http://www.gkcity.com/sitemap4.xml.gz Sitemap: http://www.gkcity.com/sitemap5.xml.gz