DataFlex

Crawling the Web

If the Chilkat Spider component only crawls a single site, how do you crawl the Web? The answer is simple: as you crawl a site, the spider collects outbound links and makes them accessible to you. You may then instantiate an instance of the Spider object for each site, and crawl it. The task of keeping track of what sites you've already crawled is left to you. This example retrieves the home page of http://www.joelonsoftware.com/ and displays the outbound links. DataFlex

Use ChilkatAx-win32.pkg

Procedure Test
    Boolean iSuccess
    Handle hoSpider
    Integer i
    String sTemp1
    Integer iTemp1

    Move False To iSuccess

    Get Create (RefClass(cComChilkatSpider)) To hoSpider
    If (Not(IsComObjectCreated(hoSpider))) Begin
        Send CreateComObject of hoSpider
    End

    // The Initialize method may be called with just the domain name,
    // such as "www.joelonsoftware.com" or a full URL.  If you pass only
    // the domain name, you must add URLs to the unspidered list by calling
    // AddUnspidered.  Otherwise, the URL you pass to Initialize is the 1st
    // URL in the unspidered list.
    Send ComInitialize To hoSpider "www.joelonsoftware.com"

    Send ComAddUnspidered To hoSpider "http://www.joelonsoftware.com/"

    Get ComCrawlNext Of hoSpider To iSuccess

    Get ComNumOutboundLinks Of hoSpider To iTemp1
    For i From 0 To (iTemp1 - 1)
        Get ComGetOutboundLink Of hoSpider i To sTemp1
        Showln sTemp1
    Loop



End_Procedure