web crawlers

基本解释

Once the Sitemap XML is completed, you can make it available to Web crawlers by submitting a URL to the sitemap to those sites that support the protocol.
完成 Sitemap XML 文件後，便可以曏支持這一協議的站點提交站點地圖的 URL，這樣 web 爬蟲程序便可以使用該文件了。
Sitemaps 0.90 is a simple and intuitive way for webmasters to provide the right level of information to Web crawlers so that they can efficiently crawl a Web site.
Sitemaps 0.90爲 Web 站點琯理員提供了簡單而且直觀的方法，使他們可以曏 web 爬蟲程序提供正確的信息，從而使爬蟲程序能夠高傚地爬行某個 Web 站點。
Also known as "invisible web", the term "deep web" refers to a vast repository of underlying content, such as documents in online databases that general-purpose web crawlers cannot reach.
“深網”也被稱做“隱形網”，指的是網絡中存儲數量浩繁的基本信息的空間，例如存儲於在線數據庫中的文件，一般用途的網絡搜索器是找不到這些文件的。

"Think algorithms and models and different software and Web crawlers, " says Christine Elliott, an American Express spokesperson.
FORBES: Magazine Article
Industry specific web crawlers or vulnerability discovery bots, network scans, and resource exhaustion attacks against web servers are rampant.
FORBES: Clean That Network Traffic
Through the use of proprietary Web crawlers, this news organization had breached the firewall to provide its readers with an upper hand.
FORBES: Data Security: A Shareholder Protection Imperative

網絡爬蟲
Web page lifetime analysis from users perspective is important to enhance the performance of web crawlers and search engines,and to improve the efficiency of web advertising.
從用戶的角度對網頁的生命周期進行分析可以提高網絡爬蟲和搜索引擎的性能,改善網絡廣告的傚果。