Web spiders — crawlers, bots, ants, scutters, whatever the vendor calls them — are automated programs that crawl web pages to extract or scrape information. Businesses use them for fraud screening, competitive intelligence, financial research, and basic link validation.
Legitimate and malicious uses
On the legitimate side: search engine indexing, link verification, competitive analysis, market research. On the malicious side: harvesting email addresses for spam lists, and copying site content wholesale to run pharming scams.
Implementation options
Build a proprietary spider in Java or PHP, buy commercial software (anywhere from under $100 to several thousand dollars), or hire a specialized monitoring provider. Manual checks — domain registration verification, WHOIS lookups, direct content inspection — are a lower-tech alternative.
Merchant website monitoring, operationally
Third-party monitoring services combine automated scanning with human review to confirm merchant compliance with Visa's Global Brand Protection Program and MasterCard's BRAM requirements — watching for illegal goods, counterfeit products, prohibited gambling, and pricing patterns that look like a scam storefront.
What to check when analyzing results
Whether the extracted content actually looks authentic, all hyperlinks resolve correctly, the HTML itself is valid, and subdomain pages get examined too — not just the homepage. This takes real internal analysis time unless you outsource to a dedicated monitoring vendor.