Visual Basic 6.0
Spider Examples for Visual Basic 6.0
Getting Started
- Getting Started Spidering a Site
- A Simple Web Crawler
- Fetch robots.txt for a Site
- Extract HTML Title, Description, Keywords
URL Utilities and Caching
- GetBaseDomain
- CanonicalizeUrl
- Using the Disk Cache
Controlling What Gets Crawled
- Avoid URLs Matching Any of a Set of Patterns
- Avoiding Outbound Links Matching Patterns
- Must-Match Patterns
- Setting a Maximum URL Length
- Setting a Maximum Response Size