Get Base Domains

Demonstrates how to accumulate a list of unique domain names referenced from outbound URLs.

Download Chilkat Spider ActiveX

Dim fso, outFile
Set fso = CreateObject("Scripting.FileSystemObject")
Set outFile = fso.CreateTextFile("output.txt", True)

' The Chilkat Spider component/library is free.
set spider = CreateObject("Chilkat.Spider")

set domainList = CreateObject("Chilkat.CkStringArray")

' Set the Unique property so that duplicates are not added.
domainList.Unique = 1

' Crawl the home page of joelonsoftware.com and get the outbound URLs
spider.Initialize "www.joelonsoftware.com"
spider.AddUnspidered "http://www.joelonsoftware.com/"

success = spider.CrawlNext()

' Build a list of unique domains.

For i = 0 To spider.NumOutboundLinks - 1
    url = spider.GetOutboundLink(i)
    domainList.Append spider.GetDomain(url)
Next

' Display the domains.
For i = 0 To domainList.Count - 1
    outFile.WriteLine(domainList.GetString(i))
    outFile.WriteLine(spider.GetBaseDomain(domainList.GetString(i)) _
         & vbCrLf)
Next
outFile.Close