Get Base Domains

Demonstrates how to accumulate a list of unique domain names referenced from outbound URLs.

Download Chilkat Spider ActiveX

LOCAL loSpider
LOCAL loDomainList
LOCAL lnSuccess
LOCAL i
LOCAL lcUrl

* The Chilkat Spider component/library is free.
loSpider = CreateObject('Chilkat.Spider')

loDomainList = CreateObject('Chilkat.CkStringArray')

* Set the Unique property so that duplicates are not added.
loDomainList.Unique = 1

* Crawl the home page of joelonsoftware.com and get the outbound URLs
loSpider.Initialize("www.joelonsoftware.com")
loSpider.AddUnspidered("http://www.joelonsoftware.com/")

lnSuccess = loSpider.CrawlNext()

* Build a list of unique domains.

FOR i = 0 TO loSpider.NumOutboundLinks - 1
    lcUrl = loSpider.GetOutboundLink(i)
    loDomainList.Append(loSpider.GetDomain(lcUrl))
NEXT

* Display the domains.
FOR i = 0 TO loDomainList.Count - 1
    ? loDomainList.GetString(i)
    ? loSpider.GetBaseDomain(loDomainList.GetString(i)) + CHR(13)+CHR(10)
NEXT