www robot:

Easy to say "robots (bots) are all internet super spies"
They can copy your websites information without notice
you(or webmaster).

Robot: a kind of soft program which deigned
to traverse (automatic download) the internet
Web hypertext structure by retrieving a 
document, and recursively retrieving all 
documents that are referenced. 

Because robots is reading hypertext ;
therefore, all robots can be recognized
by agent, server, ip.....

But you still can not reject all robots,
as log as your website is design of
hypertext structure 
because there are to many bots...

You shall be aware:: 
(Do not count on robots.txt)
It means; you shall not use robots.txt
to disallow bots to read your TOP SECRET PAGE

Your TOP SECRET PAGE shall have 
guard or security to protect but 
not only use robots disallow to avoid readers.

Ezer will consider this is a super big hole
for internet world.

The other way to think; it is also easy 
to create a robot which only catch 
all disallow links to generate a search engine
you may call it "disallow seach zone"

Attention: Not all the bots follow

the same rule to read or to run robots.txt

However, general information you can find

at http://www.robotstxt.org/wc/robots.html

and http://www.w3.org/TR/html4/struct/links.html