Chilkat
HOME
Android™
ASP
Visual Basic
VB.NET
C#
iOS (IPhone)
Objective-C
C++
C
MFC
Delphi
FoxPro
Java
Perl
PHP Extension
PHP ActiveX
Python
PowerShell
Ruby
SQL Server
VBScript
Crawling the Web
If the Chilkat Spider component only crawls a single site, how do you crawl the Web? The answer is simple: as you crawl a site, the spider collects outbound links and makes them accessible to you. You may then instantiate an instance of the Spider object for each site, and crawl it. The task of keeping track of what sites you've already crawled is left to you (for now). This example retrieves the home page of http://www.joelonsoftware.com/ and displays the outbound links. ' The Chilkat Spider component/library is free. Dim spider As New Chilkat.Spider() ' The Initialize method may be called with just the domain name, ' such as "www.joelonsoftware.com" or a full URL. If you pass only ' the domain name, you must add URLs to the unspidered list by calling ' AddUnspidered. Otherwise, the URL you pass to Initialize is the 1st ' URL in the unspidered list. spider.Initialize("www.joelonsoftware.com") spider.AddUnspidered("http://www.joelonsoftware.com/") Dim success As Boolean success = spider.CrawlNext() Dim i As Integer For i = 0 To spider.NumOutboundLinks - 1 TextBox1.Text = TextBox1.Text & spider.GetOutboundLink(i) & vbCrLf TextBox1.Refresh() Next |
© 2000-2012 Chilkat Software, Inc. All Rights Reserved.