Chilkat
HOME
Android™
ASP
Visual Basic
VB.NET
C#
iOS (IPhone)
Objective-C
C++
C
MFC
Delphi
FoxPro
Java
Perl
PHP Extension
PHP ActiveX
Python
PowerShell
Ruby
SQL Server
VBScript
|
Fetch robots.txt for a SiteThe Chilkat Spider library is robots.txt compliant. It automatically fetches a site's robots.txt file and adheres to it. It will not download pages denied by robots.txt. Pages excluded by robots.txt will not appear in the Spider's "unspidered" list. This example shows how to explicitly download and review the robots.txt for a given site.
import chilkat # The Chilkat Spider component/library is free. spider = chilkat.CkSpider() spider.Initialize("www.chilkatsoft.com") robotsText = spider.fetchRobotsText() print robotsText |
© 2000-2010 Chilkat Software, Inc. All Rights Reserved.