(Delphi ActiveX) Fetch robots.txt for a Site

The Chilkat Spider library is robots.txt compliant. It automatically fetches a site's robots.txt file and adheres to it. It will not download pages denied by robots.txt. Pages excluded by robots.txt will not appear in the Spider's "unspidered" list. This example shows how to explicitly download and review the robots.txt for a given site.

Chilkat for Delphi Downloads Chilkat ActiveX DLL for Delphi Chilkat non-ActiveX DLL for Delphi * The examples here use the ActiveX DLL.

uses
    Winapi.Windows, Winapi.Messages, System.SysUtils, System.Variants, System.Classes, Vcl.Graphics,
    Vcl.Controls, Vcl.Forms, Vcl.Dialogs, Vcl.StdCtrls, Chilkat_TLB;

...

procedure TForm1.Button1Click(Sender: TObject);
var
spider: TChilkatSpider;
robotsText: WideString;

begin
spider := TChilkatSpider.Create(Self);

spider.Initialize('www.chilkatsoft.com');

robotsText := spider.FetchRobotsText();

Memo1.Lines.Add(robotsText);
end;