MFC Examples

ChilkatHOMEAndroid™ASPVisual BasicVB.NETC#iOS (IPhone)Objective-CC++CMFCDelphiFoxProJavaPerl
PHP ExtensionPHP ActiveXPythonPowerShellRubySQL ServerVBScript

MFC Examples

Bounced Mail
Bz2
Certificates/Keys
Charset
CSV
Diffie-Hellman
DKIM / DomainKey
DSA
Email Object
Encryption
FileAccess
FTP
HTML Conversion
HTTP
IMAP
MHT / HTML Email
MIME
NTLM
POP3
RSA
SMTP
Socket
Spider
SSH Key
SSH
SSH Tunnel
SFTP
Tar
Upload
XML
Zip
Amazon S3


 

 

 

 

 

 

 

 

Fetch robots.txt for a Site

The Chilkat Spider library is robots.txt compliant. It automatically fetches a site's robots.txt file and adheres to it. It will not download pages denied by robots.txt. Pages excluded by robots.txt will not appear in the Spider's "unspidered" list. This example shows how to explicitly download and review the robots.txt for a given site.

Downloads:

MS Windows Visual C/C++ Libraries
Linux/CentOS C/C++ Libraries
MAC OS X C/C++ Libraries
Solaris C/C++ Libraries
C++ Builder Libraries
// Needs #include <CkSpider.h>

    CkString strOut;

    //  The Chilkat Spider component/library is free.
    CkSpider spider;

    spider.Initialize("www.chilkatsoft.com");

    const char * robotsText;
    robotsText = spider.fetchRobotsText();

    strOut.append(robotsText);
    strOut.append("\r\n");

    SetDlgItemText(IDC_EDIT1,strOut.getUnicode());

Need a specific example? Send a request to support@chilkatsoft.com

© 2000-2010 Chilkat Software, Inc. All Rights Reserved.