Setting a Maximum Response Size

The MaxResponseSize property protects your spider from downloading a page that is too large. By default, MaxResponseSize = 0, which indicates that there is no maximum. You may set it to a number indicating the maximum number of bytes to download. URLs with response sizes larger than this will be skipped.

Download Chilkat Spider ActiveX

uses
    Windows, Messages, SysUtils, Variants, Classes, Graphics, Controls, Forms,
    Dialogs, StdCtrls,
    SPIDERXLib_TLB,
    SPIDERXLib_TLB,
    OleCtrls;

...

procedure TForm1.Button1Click(Sender: TObject);
var
spider: TSpider;

begin
//  The Chilkat Spider component/library is free.
spider := TSpider.Create(Self);

spider.Initialize('www.chilkatsoft.com');

//  Add the 1st URL:
spider.AddUnspidered('http://www.chilkatsoft.com/');

//  This example demonstrates setting the MaxResponseSize property
//  Do not download anything with a response size greater than 100,000 bytes.
spider.MaxResponseSize := 100000;
end;