List of web pages that the search engine can crawl, i.e. all pages on the web that allow access to spiders.