Get Referenced Domains

Demonstrates how to accumulate a list of unique domain names referenced from outbound URLs.

LOCAL loSpider
LOCAL loDomainList
LOCAL lnSuccess
LOCAL i
LOCAL lcUrl

*  The Chilkat Spider component/library is free.
loSpider = CreateObject('Chilkat.Spider')

loDomainList = CreateObject('Chilkat.CkStringArray')

*  Set the Unique property so that duplicates are not added.
loDomainList.Unique = 1

*  Crawl the home page of joelonsoftware.com and get the outbound URLs
loSpider.Initialize("www.joelonsoftware.com")
loSpider.AddUnspidered("http://www.joelonsoftware.com/")

lnSuccess = loSpider.CrawlNext()

*  Build a list of unique domains.

FOR i = 0 TO loSpider.NumOutboundLinks - 1
    lcUrl = loSpider.GetOutboundLink(i)
    loDomainList.Append(loSpider.GetDomain(lcUrl))
NEXT

*  Display the domains.
FOR i = 0 TO loDomainList.Count - 1
    ? loDomainList.GetString(i)
NEXT