Skip to main content
Select Source:

Spiders

SPIDERS

When search engines (like Yahoo or Alta Vista, for example) are used to find information on the Internet, the results one receives normally come from giant indexes or databases, instead of from the actual Internet in real time. Because the Internet changes constantly, a search engine's index must be continually updated. Spiders are the tools used for accomplishing this critical task. They work in tandem with indexes and search software to comb the Web for information. Without spiders, it would be difficult to find new Web sites, or current content on existing ones. Also called crawlers, ants, or wanderers, spiders technically are a member of the bot familysoftware programs that operate unattended, usually on the Internet. Therefore, spiders often are referred to as bots. However, it's important to know that spiders are not the same as intelligent agentsanother kind of bot that has a wider range of capabilities, including interactivity.

Spiders travel from server to server, visiting different areas of the Internetnormally sites on the World Wide Web but also File Transfer Protocol (FTP) sites and Gopher archives. This process, known as discovery, can be performed blindly or in a more directed manner. When done blindly, a spider attempts to visit every possible Internet Protocol (IP) addressunique numbers assigned to every machine on the Internet. This approach takes longer than a directed approach, which involves searching registered domain names, or the names used to identify a site (such as Intel.com). Both approaches have advantages and drawbacks. Large search engines often employ many spiders at once, working in parallel on many different machines or servers to archive the online world in one database. After spiders report back to search engines with new information, it often takes additional time before the information is updated in the engine's index and made available for end-users to see in search results.

Some spiders record every word on a Web site for their respective indexes, while others only report certain keywords listed in meta tags. Although they usually aren't visible to someone using a Web browser, meta tags are special codes that provide keywords or Web site descriptions to spiders. Sometimes, the information listed in meta tags is incorrect or misleading, which causes spiders to deliver inaccurate descriptions of Web sites to indexes. In any case, the issue of keywords and how they are placed, either within actual Web site content or in meta tags, is important to online marketers. The majority of consumers reach e-commerce sites through search engines, and the right keywords increase the odds a company's site will be included in search results.

While spiders are critical elements of the online world, they also were a source of aggravation and controversy in the early 2000s. On the technical side, spiders sometimes slow down the performance of Web serversthe computers or applications that host Web sitesby visiting them over and over in a short period of time, sometimes as often as 100 times in a single minute. An example of this type of behavior includes spiders that search for up-to-the-minute news, or product or stock-market information. For companies without strong technical systems, spiders that exhibit this kind of behavior can cause major problems.

Another concern centered around how information collected by spiders was gathered, redistributed to other parties, and ultimately used. Part of this concern, which created related legal issues, involved security, because spiders sometimes uncover information a site's owner considers private or off-limits to the public. Another issue was misrepresentation, especially for distributors who risked having old, incorrect information about product availability or inventory displayed on other Web sites.

Because of these concerns, Web site administrators took measures to deny spiders access to their Web sites, or to certain areas within them. Administrators post specific rules about what spiders are allowed to access on their sites in an exclusion file called robots.txt, which spiders normally find and read. These rules can also be seen by the naked eye if one adds robots.txt to the end of a site's address or uniform resource locator (URL). By looking at an access log, site administrators are able to determine which spiders have visited their sites and what information they recorded. In this situation, spiders can be identified individually by name, which are given to them by their creators. This gives administrators the ability to exclude certain bots from visiting in the future, should they present a problem.

FURTHER READING:

Baljko, Shah. "Web Crawling: Sticky Issue For Distributors." Planet IT, November 1, 2000. Available from www.PlanetIT.com.

Champlin, Leslie. "E-firms Lure Search Spiders to Their Corner of the Web." The Business Journal of Kansas City, March 31, 2000. Available from www.kansascity.bcentral.com.

"Crawler." Tech Encyclopedia, February 1, 2001. Available from www.techweb.com/encyclopedia.

"How Search Engines Work." Search Engine Watch, February 1, 2001. Available from www.searchenginewatch.com.

Pallmann, David. Programming Bots, Spiders, and Intelligent Agents in Microsoft Visual C++. Redmond, Washington: Microsoft Press. 1999.

"Spider." Ecommerce Webopedia, February 1, 2001. Available from e-comm.webopedia.com.

"Spider." Netlingo, January 31, 2001. Available from www.netlingo.com/inframes.

"SpiderSpotting." Search Engine Watch, February 1, 2001. Available from searchenginewatch.internet.com.

"What's a Bot?" Internet.com , February 6, 2001. Available from www.bots.internet.com.

SEE ALSO: Intelligent Agents; Results Ranking; Search Engine Strategy; Webcrawler

Cite this article
Pick a style below, and copy the text for your bibliography.

  • MLA
  • Chicago
  • APA

"Spiders." Gale Encyclopedia of E-Commerce. . Encyclopedia.com. 17 Aug. 2017 <http://www.encyclopedia.com>.

"Spiders." Gale Encyclopedia of E-Commerce. . Encyclopedia.com. (August 17, 2017). http://www.encyclopedia.com/economics/encyclopedias-almanacs-transcripts-and-maps/spiders

"Spiders." Gale Encyclopedia of E-Commerce. . Retrieved August 17, 2017 from Encyclopedia.com: http://www.encyclopedia.com/economics/encyclopedias-almanacs-transcripts-and-maps/spiders

spider

spider, organism, mostly terrestrial, of the class Arachnida, order Araneae, with four pairs of legs and a two-part body consisting of a cephalothorax, or prosoma, and an unsegmented abdomen, or opisthosoma.

The cephalothorax is covered by a shield, or carapace, and bears eight simple eyes. On the underside of the head (the cephalic part of the cephalothorax) are two pairs of appendages, the anterior pair called chelicerae and the second pair pedipalps, with which the spider captures and paralyzes its prey, injecting into it venom produced in the poison glands. The spider then liquefies the tissues of the prey with a digestive fluid and sucks this broth into its stomach where it may be stored in a digestive gland.

Breathing is by means of tracheae (air tubes) or book lungs, or both. Arachnid book lungs are similar to the gill books of horseshoe crabs but are internal and adapted to a terrestrial habitat. Young, growing spiders can regenerate missing legs and parts of legs.

Three pairs of spinnerets toward the tip of the abdomen produce protein-containing fluids that harden as they are drawn out to form silk threads. Several kinds of silk glands and spinnerets produce different kinds of silk used variously for constructing cocoons or egg sacs, spinning webs, and binding prey; other light strands are spun out for ballooning, or floating, the spiders, especially young ones, long distances on air currents. Spider silk is used for the cross hairs in certain optical instruments.

Spiders live chiefly on insects and other arthropods; some large spiders ensnare and kill small snakes, birds, and mammals. Many are cannibalistic; the female may eat the male when courtship and mating are completed. Most species are solitary, but a few live socially. Several species of spiders have bites that are exceptionally painful, or even dangerous to humans. Species of black widow spiders, which are found in the warmer parts of the world including the United States and S Canada, have a virulent neurotoxic venom. The bite venom of the brown recluse spider of SE and S central United States decomposes tissue, resulting in slow healing and sometimes leaving a sunken scar as large as a quarter.

Among the more interesting spiders are the tarantula; the trapdoor spider, which ambushes its prey from a silk-lined burrow covered by a hinged lid; the orb weavers, which spin beautiful circular webs; the diving bell spider, which lives underwater and uses a silk-enclosed air bubble to breathe; and the crab spider, jumping spider, and wolf spider, named for their habits. Spiders are classified in the phylum Arthropoda, class Arachnida, order Araneae.

See B. J. Kaston, How to Know the Spiders (3d ed. 1978); R. F. Foelix, Biology of Spiders (1982); The Audubon Society Field Guide to North American Insects and Spiders (1992).

Cite this article
Pick a style below, and copy the text for your bibliography.

  • MLA
  • Chicago
  • APA

"spider." The Columbia Encyclopedia, 6th ed.. . Encyclopedia.com. 17 Aug. 2017 <http://www.encyclopedia.com>.

"spider." The Columbia Encyclopedia, 6th ed.. . Encyclopedia.com. (August 17, 2017). http://www.encyclopedia.com/reference/encyclopedias-almanacs-transcripts-and-maps/spider

"spider." The Columbia Encyclopedia, 6th ed.. . Retrieved August 17, 2017 from Encyclopedia.com: http://www.encyclopedia.com/reference/encyclopedias-almanacs-transcripts-and-maps/spider

Spider

Spider

Various folklore beliefs surround the spider. In England, spiders were known as "money makers." If found on clothing, they were a sign that money was on the way, provided that the spider was not killed. A similar idea prevailed in Polynesia, where a spider dropping down in front of a person was a sign of a present. An American belief is that killing a spider will bring rain.

In folk medicine, a spider was rolled in butter or molasses and swallowed. As a cure for ague, it was tied up and secured on the left arm. A spider was also traditionally used as an amulet. The insect was baked and worn around the neck.

The British antiquary Elias Ashmole stated in his Memoirs (1717): "I took early in the morning a good dose of elixir, and hung three spiders around my neck, and they drove my ague away. Deo Gratias!"

Robert Burton (1577-1640) stated:

"Being in the country in the vacation time, not many years since, at Lindly in Leicestershire, my father's house, I first observed this amulet of a spider in a nut-shell, wrapped in silk, so applied for an ague by my mother. This I thought most absurd and ridiculous, and I could see no warrant in it till at length, rambling amongst authors, I found this very medicine in Dioscorides, approved by Matthiolus, repeated by Aldrovandus. I began to have a better opinion of it, and to give more credit to amulets, when I saw it in some parties answer to experience."

Spiders were sacred to the ancient Egyptian goddess Maat and are used today as symbolism of a Maatian (feminist) form of ceremonial magic.

Cite this article
Pick a style below, and copy the text for your bibliography.

  • MLA
  • Chicago
  • APA

"Spider." Encyclopedia of Occultism and Parapsychology. . Encyclopedia.com. 17 Aug. 2017 <http://www.encyclopedia.com>.

"Spider." Encyclopedia of Occultism and Parapsychology. . Encyclopedia.com. (August 17, 2017). http://www.encyclopedia.com/science/encyclopedias-almanacs-transcripts-and-maps/spider

"Spider." Encyclopedia of Occultism and Parapsychology. . Retrieved August 17, 2017 from Encyclopedia.com: http://www.encyclopedia.com/science/encyclopedias-almanacs-transcripts-and-maps/spider

spider

spider in proverbial and traditional allusion, references are made to the cunning, skill, and industry of the spider, as well as its power of secreting or emitting poison. In a traditional story of the Scottish nationalist leader Robert the Bruce (1274–1329, from 1301 Robert I of Scotland) encouraged by watching a spider in a cave attempting to spin a thread long enough to reach another piece of rock, it becomes a type of perseverance.

A spider is also said to have helped Muhammad. According to the story, the Prophet and his friend Abu-Bakr were in flight from the men of Mecca and took shelter in a cave; to protect them, pigeons built their nests and a spider spun its web across the mouth of the cave, so that the pursuers assumed that it was undisturbed.

In Greek mythology, the weaver Arachne who challenged Athene to a contest of skill was changed into a spider by the angry goddess.

See also if you want to live and thrive, let the spider run alive.

Cite this article
Pick a style below, and copy the text for your bibliography.

  • MLA
  • Chicago
  • APA

"spider." The Oxford Dictionary of Phrase and Fable. . Encyclopedia.com. 17 Aug. 2017 <http://www.encyclopedia.com>.

"spider." The Oxford Dictionary of Phrase and Fable. . Encyclopedia.com. (August 17, 2017). http://www.encyclopedia.com/humanities/dictionaries-thesauruses-pictures-and-press-releases/spider

"spider." The Oxford Dictionary of Phrase and Fable. . Retrieved August 17, 2017 from Encyclopedia.com: http://www.encyclopedia.com/humanities/dictionaries-thesauruses-pictures-and-press-releases/spider

spider

spi·der / ˈspīdər/ • n. an eight-legged predatory arachnid (order Araneae, class Arachnida) with an unsegmented body consisting of a fused head and thorax and a rounded abdomen. Spiders have fangs that inject poison into their prey, and most kinds spin webs in which to capture insects. ∎  used in names of similar or related arachnids, e.g., sea spider. ∎  any object resembling a spider, esp. one having numerous or prominent legs or radiating spokes. ∎  a cast-iron iron frying pan, originally made with legs for cooking on coals in a hearth. • v. [intr.] move in a scuttling manner suggestive of a spider: a nuthatch spidered head first down the tree trunk. ∎  form a pattern suggestive of a spider or its web. DERIVATIVES: spi·der·ish adj.

Cite this article
Pick a style below, and copy the text for your bibliography.

  • MLA
  • Chicago
  • APA

"spider." The Oxford Pocket Dictionary of Current English. . Encyclopedia.com. 17 Aug. 2017 <http://www.encyclopedia.com>.

"spider." The Oxford Pocket Dictionary of Current English. . Encyclopedia.com. (August 17, 2017). http://www.encyclopedia.com/humanities/dictionaries-thesauruses-pictures-and-press-releases/spider-1

"spider." The Oxford Pocket Dictionary of Current English. . Retrieved August 17, 2017 from Encyclopedia.com: http://www.encyclopedia.com/humanities/dictionaries-thesauruses-pictures-and-press-releases/spider-1

spiders

spiders
1. See ARACHNIDA; ARANEAE; LIPHISTIIDAE; MIMETIDAE; SPARASSIDAE; TRAPDOOR SPIDERS.

2. cobweb (comb-footed) spiders See THERIDIIDAE.

3. crab spiders See THOMISIDAE.

4. funnel-web spiders, tarantulas See MYGALOMORPHAE; THERAPHOSIDAE.

5. hunting (wolf) spiders See LYCOSIDAE.

6. jumping spiders See SALTICIDAE.

7. money spiders See LINYPHIIDAE.

8. nursery-web spiders See PISAURIDAE.

9. orb-web spiders See ARANEIDAE.

10. purse-web spiders See ATYPIDAE.

11. sheet-web spiders See AGELENIDAE.

12. spitting spiders See SCYTODIDAE.

Cite this article
Pick a style below, and copy the text for your bibliography.

  • MLA
  • Chicago
  • APA

"spiders." A Dictionary of Zoology. . Encyclopedia.com. 17 Aug. 2017 <http://www.encyclopedia.com>.

"spiders." A Dictionary of Zoology. . Encyclopedia.com. (August 17, 2017). http://www.encyclopedia.com/science/dictionaries-thesauruses-pictures-and-press-releases/spiders-0

"spiders." A Dictionary of Zoology. . Retrieved August 17, 2017 from Encyclopedia.com: http://www.encyclopedia.com/science/dictionaries-thesauruses-pictures-and-press-releases/spiders-0

spider

spider Any of numerous species of terrestrial, invertebrate, arachnid arthropods found throughout the world in a wide variety of habitats. Spiders have an unsegmented abdomen attached to a cephalothorax by a slender pedicel. There are no antennae; sensory hairs are found on the appendages (four pairs of walking legs). Most species have spinnerets on the abdomen for spinning silk to make egg cases and webs.

Cite this article
Pick a style below, and copy the text for your bibliography.

  • MLA
  • Chicago
  • APA

"spider." World Encyclopedia. . Encyclopedia.com. 17 Aug. 2017 <http://www.encyclopedia.com>.

"spider." World Encyclopedia. . Encyclopedia.com. (August 17, 2017). http://www.encyclopedia.com/environment/encyclopedias-almanacs-transcripts-and-maps/spider

"spider." World Encyclopedia. . Retrieved August 17, 2017 from Encyclopedia.com: http://www.encyclopedia.com/environment/encyclopedias-almanacs-transcripts-and-maps/spider

spider

spider (crawler, Web crawler) An automatic program that searches the Internet, finding new Web sites and producing an index of addresses and content for use in a search engine.

Cite this article
Pick a style below, and copy the text for your bibliography.

  • MLA
  • Chicago
  • APA

"spider." A Dictionary of Computing. . Encyclopedia.com. 17 Aug. 2017 <http://www.encyclopedia.com>.

"spider." A Dictionary of Computing. . Encyclopedia.com. (August 17, 2017). http://www.encyclopedia.com/computing/dictionaries-thesauruses-pictures-and-press-releases/spider

"spider." A Dictionary of Computing. . Retrieved August 17, 2017 from Encyclopedia.com: http://www.encyclopedia.com/computing/dictionaries-thesauruses-pictures-and-press-releases/spider

spider

spider OE. spīðra :- *spinþran-, f. spinnan SPIN.

Cite this article
Pick a style below, and copy the text for your bibliography.

  • MLA
  • Chicago
  • APA

"spider." The Concise Oxford Dictionary of English Etymology. . Encyclopedia.com. 17 Aug. 2017 <http://www.encyclopedia.com>.

"spider." The Concise Oxford Dictionary of English Etymology. . Encyclopedia.com. (August 17, 2017). http://www.encyclopedia.com/humanities/dictionaries-thesauruses-pictures-and-press-releases/spider-2

"spider." The Concise Oxford Dictionary of English Etymology. . Retrieved August 17, 2017 from Encyclopedia.com: http://www.encyclopedia.com/humanities/dictionaries-thesauruses-pictures-and-press-releases/spider-2

spiders

spiders See CHELICERATA.

Cite this article
Pick a style below, and copy the text for your bibliography.

  • MLA
  • Chicago
  • APA

"spiders." A Dictionary of Earth Sciences. . Encyclopedia.com. 17 Aug. 2017 <http://www.encyclopedia.com>.

"spiders." A Dictionary of Earth Sciences. . Encyclopedia.com. (August 17, 2017). http://www.encyclopedia.com/science/dictionaries-thesauruses-pictures-and-press-releases/spiders

"spiders." A Dictionary of Earth Sciences. . Retrieved August 17, 2017 from Encyclopedia.com: http://www.encyclopedia.com/science/dictionaries-thesauruses-pictures-and-press-releases/spiders

spiders

spiders See Arachnida.

Cite this article
Pick a style below, and copy the text for your bibliography.

  • MLA
  • Chicago
  • APA

"spiders." A Dictionary of Biology. . Encyclopedia.com. 17 Aug. 2017 <http://www.encyclopedia.com>.

"spiders." A Dictionary of Biology. . Encyclopedia.com. (August 17, 2017). http://www.encyclopedia.com/science/dictionaries-thesauruses-pictures-and-press-releases/spiders-1

"spiders." A Dictionary of Biology. . Retrieved August 17, 2017 from Encyclopedia.com: http://www.encyclopedia.com/science/dictionaries-thesauruses-pictures-and-press-releases/spiders-1

spider

spiderbidder, consider, Jiddah, kidder, whydah •bewilder, builder, guilder, Hilda, Matilda, St Kilda, Tilda, tilde •Belinda, Cabinda, cinder, Clarinda, Dorinda, hinder, Kinder, Linda, Lucinda, Melinda, tinder •Drogheda • shipbuilder • bodybuilder •coachbuilder • boatbuilder • Candida •spina bifida •calendar, calender •Phillida • cylinder • Phasmida •Andromeda • Mérida • Florida •Cressida • lavender • provender •chider, cider, divider, eider, glider, Guider, Haida, hider, Ida, insider, Oneida, outsider, provider, rider, Ryder, Saida, slider, spider, strider, stridor •Wilder •binder, blinder, finder, grinder, kinda, minder, ringbinder, winder •Fassbinder • spellbinder • highbinder •bookbinder • pathfinder •rangefinder • viewfinder • backslider •paraglider • childminder • outrider •joyrider • roughrider • ringsider •Tynesider • sidewinder

Cite this article
Pick a style below, and copy the text for your bibliography.

  • MLA
  • Chicago
  • APA

"spider." Oxford Dictionary of Rhymes. . Encyclopedia.com. 17 Aug. 2017 <http://www.encyclopedia.com>.

"spider." Oxford Dictionary of Rhymes. . Encyclopedia.com. (August 17, 2017). http://www.encyclopedia.com/humanities/dictionaries-thesauruses-pictures-and-press-releases/spider-0

"spider." Oxford Dictionary of Rhymes. . Retrieved August 17, 2017 from Encyclopedia.com: http://www.encyclopedia.com/humanities/dictionaries-thesauruses-pictures-and-press-releases/spider-0