Fetch robots.txt for a Site

The Chilkat Spider library is robots.txt compliant. It automatically fetches a site's robots.txt file and adheres to it. It will not download pages denied by robots.txt. Pages excluded by robots.txt will not appear in the Spider's "unspidered" list. This example shows how to explicitly download and review the robots.txt for a given site.

Download Chilkat C/C++ Libraries for VC++ 9.0 / Win32

Download Chilkat C/C++ Libraries for VC++ 8.0 / Win32

Download Chilkat C/C++ 64-bit Libraries for VC++ 8.0 / x64

Download Chilkat Visual Studio 2005 C/C++ Libs for Windows Mobile, Pocket PC, SmartPhone, WinCE

Download Chilkat C/C++ Libraries for VC++ 7.0 / Win32

Download Chilkat C/C++ Libraries for VC++ 6.0 / Win32

Download Chilkat C/C++ Libraries for VC++ 6.0, Win 95/98/NT4 Compatible

// Needs #include <CkSpider.h>

    CkString strOut;

    //  The Chilkat Spider component/library is free.
    CkSpider spider;

    spider.Initialize("www.chilkatsoft.com");

    const char * robotsText;
    robotsText = spider.fetchRobotsText();

    strOut.append(robotsText);
    strOut.append("\r\n");

    SetDlgItemText(IDC_EDIT1,strOut.getUnicode());