When search engines (like Yahoo or Alta Vista, for example) are used to find information on the Internet, the results one receives normally come from giant indexes or databases, instead of from the actual Internet in real time. Because the Internet changes constantly, a search engine's index must be continually updated. Spiders are the tools used for accomplishing this critical task. They work in tandem with indexes and search software to comb the Web for information. Without spiders, it would be difficult to find new Web sites, or current content on existing ones. Also called crawlers, ants, or wanderers, spiders technically are a member of the bot family—software programs that operate unattended, usually on the Internet. Therefore, spiders often are referred to as bots. However, it's important to know that spiders are not the same as intelligent agents—another kind of bot that has a wider range of capabilities, including interactivity.
Spiders travel from server to server, visiting different areas of the Internet—normally sites on the World Wide Web but also File Transfer Protocol (FTP) sites and Gopher archives. This process, known as discovery, can be performed blindly or in a more directed manner. When done blindly, a spider attempts to visit every possible Internet Protocol (IP) address—unique numbers assigned to every machine on the Internet. This approach takes longer than a directed approach, which involves searching registered domain names, or the names used to identify a site (such as Intel.com). Both approaches have advantages and drawbacks. Large search engines often employ many spiders at once, working in parallel on many different machines or servers to archive the online world in one database. After spiders report back to search engines with new information, it often takes additional time before the information is updated in the engine's index and made available for end-users to see in search results.
Some spiders record every word on a Web site for their respective indexes, while others only report certain keywords listed in meta tags. Although they usually aren't visible to someone using a Web browser, meta tags are special codes that provide keywords or Web site descriptions to spiders. Sometimes, the information listed in meta tags is incorrect or misleading, which causes spiders to deliver inaccurate descriptions of Web sites to indexes. In any case, the issue of keywords and how they are placed, either within actual Web site content or in meta tags, is important to online marketers. The majority of consumers reach e-commerce sites through search engines, and the right keywords increase the odds a company's site will be included in search results.
While spiders are critical elements of the online world, they also were a source of aggravation and controversy in the early 2000s. On the technical side, spiders sometimes slow down the performance of Web servers—the computers or applications that host Web sites—by visiting them over and over in a short period of time, sometimes as often as 100 times in a single minute. An example of this type of behavior includes spiders that search for up-to-the-minute news, or product or stock-market information. For companies without strong technical systems, spiders that exhibit this kind of behavior can cause major problems.
Another concern centered around how information collected by spiders was gathered, redistributed to other parties, and ultimately used. Part of this concern, which created related legal issues, involved security, because spiders sometimes uncover information a site's owner considers private or off-limits to the public. Another issue was misrepresentation, especially for distributors who risked having old, incorrect information about product availability or inventory displayed on other Web sites.
Because of these concerns, Web site administrators took measures to deny spiders access to their Web sites, or to certain areas within them. Administrators post specific rules about what spiders are allowed to access on their sites in an exclusion file called robots.txt, which spiders normally find and read. These rules can also be seen by the naked eye if one adds robots.txt to the end of a site's address or uniform resource locator (URL). By looking at an access log, site administrators are able to determine which spiders have visited their sites and what information they recorded. In this situation, spiders can be identified individually by name, which are given to them by their creators. This gives administrators the ability to exclude certain bots from visiting in the future, should they present a problem.
Baljko, Shah. "Web Crawling: Sticky Issue For Distributors." Planet IT, November 1, 2000. Available from www.PlanetIT.com.
Champlin, Leslie. "E-firms Lure Search Spiders to Their Corner of the Web." The Business Journal of Kansas City, March 31, 2000. Available from www.kansascity.bcentral.com.
"Crawler." Tech Encyclopedia, February 1, 2001. Available from www.techweb.com/encyclopedia.
"How Search Engines Work." Search Engine Watch, February 1, 2001. Available from www.searchenginewatch.com.
Pallmann, David. Programming Bots, Spiders, and Intelligent Agents in Microsoft Visual C++. Redmond, Washington: Microsoft Press. 1999.
"Spider." Ecommerce Webopedia, February 1, 2001. Available from e-comm.webopedia.com.
"Spider." Netlingo, January 31, 2001. Available from www.netlingo.com/inframes.
"SpiderSpotting." Search Engine Watch, February 1, 2001. Available from searchenginewatch.internet.com.
SEE ALSO: Intelligent Agents; Results Ranking; Search Engine Strategy; Webcrawler
1. See ARACHNIDA; ARANEAE; LIPHISTIIDAE; MIMETIDAE; SPARASSIDAE; TRAPDOOR SPIDERS.
2. cobweb (comb-footed) spiders See THERIDIIDAE.
3. crab spiders See THOMISIDAE.
4. funnel-web spiders, tarantulas See MYGALOMORPHAE; THERAPHOSIDAE.
5. hunting (wolf) spiders See LYCOSIDAE.
6. jumping spiders See SALTICIDAE.
7. money spiders See LINYPHIIDAE.
8. nursery-web spiders See PISAURIDAE.
9. orb-web spiders See ARANEIDAE.
10. purse-web spiders See ATYPIDAE.
11. sheet-web spiders See AGELENIDAE.
12. spitting spiders See SCYTODIDAE.
Various folklore beliefs surround the spider. In England, spiders were known as "money makers." If found on clothing, they were a sign that money was on the way, provided that the spider was not killed. A similar idea prevailed in Polynesia, where a spider dropping down in front of a person was a sign of a present. An American belief is that killing a spider will bring rain.
In folk medicine, a spider was rolled in butter or molasses and swallowed. As a cure for ague, it was tied up and secured on the left arm. A spider was also traditionally used as an amulet. The insect was baked and worn around the neck.
The British antiquary Elias Ashmole stated in his Memoirs (1717): "I took early in the morning a good dose of elixir, and hung three spiders around my neck, and they drove my ague away. Deo Gratias!"
Robert Burton (1577-1640) stated:
"Being in the country in the vacation time, not many years since, at Lindly in Leicestershire, my father's house, I first observed this amulet of a spider in a nut-shell, wrapped in silk, so applied for an ague by my mother…. This I thought most absurd and ridiculous, and I could see no warrant in it … till at length, rambling amongst authors, I found this very medicine in Dioscorides, approved by Matthiolus, repeated by Aldrovandus…. I began to have a better opinion of it, and to give more credit to amulets, when I saw it in some parties answer to experience."
Spiders were sacred to the ancient Egyptian goddess Maat and are used today as symbolism of a Maatian (feminist) form of ceremonial magic.
spi·der / ˈspīdər/ • n. an eight-legged predatory arachnid (order Araneae, class Arachnida) with an unsegmented body consisting of a fused head and thorax and a rounded abdomen. Spiders have fangs that inject poison into their prey, and most kinds spin webs in which to capture insects. ∎ used in names of similar or related arachnids, e.g., sea spider. ∎ any object resembling a spider, esp. one having numerous or prominent legs or radiating spokes. ∎ a cast-iron iron frying pan, originally made with legs for cooking on coals in a hearth. • v. [intr.] move in a scuttling manner suggestive of a spider: a nuthatch spidered head first down the tree trunk. ∎ form a pattern suggestive of a spider or its web. DERIVATIVES: spi·der·ish adj.
A spider is also said to have helped Muhammad. According to the story, the Prophet and his friend Abu-Bakr were in flight from the men of Mecca and took shelter in a cave; to protect them, pigeons built their nests and a spider spun its web across the mouth of the cave, so that the pursuers assumed that it was undisturbed.
In Greek mythology, the weaver Arachne who challenged Athene to a contest of skill was changed into a spider by the angry goddess.
See also if you want to live and thrive, let the spider run alive.
Spider ★★★★ 2002 (R)
Gripping and well-directed psychodrama tells the story of Spider, a paranoid-schizophrenic, as he tries to resolve his convoluted memories and emotions. After being released from a mental hospital, Spider (Ralph Fiennes) lives in an eerie halfway house for mental patients. There he recounts the events of his life that led to his tortured state of mind. Yet, since the film is told entirely from Spider's perspective it's impossible to distinguish between reality and the blurred fiction of a madman, making the entire film both illusionary and fascinatingly real. Stellar performances from Fiennes, Miranda Richardson, Gabriel Byrne, and Bradley Hall in an amazing turn as the young Spider. Cronenberg's most polished drama to date shows that he's successfully outgrown the moniker of Canada's “baron of blood.” 98m/C VHS, DVD . CA GB Ralph Fiennes, Miranda Richardson, Gabriel Byrne, Bradley Hall, Lynn Redgrave, John Neville, Gary Reineke, Sara Stockbridge, Philip Craig; D: David Cronenberg; W: Patrick McGrath; C: Peter Suschitzky; M: Howard Shore. Genie '02: Director (Cronenberg).
SPIDER (Heb. עַכָּבִישׁ, akkavish). Isaiah (59:5–6) compares the evil designs of those who plot against the righteous to the webs which the spider spins to trap insects, while Job (8:14–15) compares the house of the wicked to the spider's fragile web. There are hundreds of species of spider in Israel, all having poisonous glands in their maxillaries. The poison in most spiders is a mild one, but there are species capable of killing a bird or a mouse. It would appear that the akhshuv (Ps. 140:4) which is mentioned together with the snake as a poisonous animal is merely the akkavish with the letters transposed. The Tosefta (Par. 9:6) enumerates it among the species of spiders. Some erroneously identify the spider with the semamit (Prov. 30:28) which is the *gecko.
Lewysohn, Zool, 299–301, nos. 400 and 401; F.S. Bodenheimer, Animal and Man in Bible Lands (1960), 116, nos. 336–40; J. Feliks, Animal World of the Bible (1962), 135.