Denying search engines access to your site
User-agent: * Disallow: /
When search engines first attempt to index your site they look for the robots.txt file. within this file you have the power to tell as specific or all (*) search engines that they are not allowed any directories on the site.
Specific search engines: You may also what to specify a single search engine that you want to exclude or the other way around. Allow only one search engine but not others
-- Allow Bing to index User-agent: Bing Disallow: User-agent: * Disallow: / -- Exclude only Bing User-agent: Bing Disallow: /
Exclude specifics: Using the same ideas as above you can allow and deny spcific directories and files on your server.
User-agent: * Disallow: /~some/coolThings/