Chilkat HOME .NET Core C# Android™ AutoIt C C# C++ Chilkat2-Python CkPython Classic ASP DataFlex Delphi ActiveX Delphi DLL Go Java Lianja Mono C# Node.js Objective-C PHP ActiveX PHP Extension Perl PowerBuilder PowerShell PureBasic Ruby SQL Server Swift 2 Swift 3,4,5... Tcl Unicode C Unicode C++ VB.NET VBScript Visual Basic 6.0 Visual FoxPro Xojo Plugin
(Delphi ActiveX) Fetch robots.txt for a SiteThe Chilkat Spider library is robots.txt compliant. It automatically fetches a site's robots.txt file and adheres to it. It will not download pages denied by robots.txt. Pages excluded by robots.txt will not appear in the Spider's "unspidered" list. This example shows how to explicitly download and review the robots.txt for a given site.
uses Winapi.Windows, Winapi.Messages, System.SysUtils, System.Variants, System.Classes, Vcl.Graphics, Vcl.Controls, Vcl.Forms, Vcl.Dialogs, Vcl.StdCtrls, Chilkat_v9_5_0_TLB; ... procedure TForm1.Button1Click(Sender: TObject); var spider: TChilkatSpider; robotsText: WideString; begin spider := TChilkatSpider.Create(Self); spider.Initialize('www.chilkatsoft.com'); robotsText := spider.FetchRobotsText(); Memo1.Lines.Add(robotsText); end; |
© 2000-2023 Chilkat Software, Inc. All Rights Reserved.