spider

The Web-walking part of a search engine that collects pages for indexing in the search engine’s database. Also called a bot. The best-known spider is Scooter, the web-walker for the Alta Vista search engine.