Android™

Fetch robots.txt for a Site

See more Spider Examples

The Chilkat Spider library is robots.txt compliant. It automatically fetches a site's robots.txt file and adheres to it. It will not download pages denied by robots.txt. Pages excluded by robots.txt will not appear in the Spider's "unspidered" list. This example shows how to explicitly download and review the robots.txt for a given site. Android™

// Important: Don't forget to include the call to System.loadLibrary
// as shown at the bottom of this code sample.
package com.test;

import android.app.Activity;
import com.chilkatsoft.*;

import android.widget.TextView;
import android.os.Bundle;

public class SimpleActivity extends Activity {

  private static final String TAG = "Chilkat";

  // Called when the activity is first created.
  @Override
  public void onCreate(Bundle savedInstanceState) {
    super.onCreate(savedInstanceState);

    CkSpider spider = new CkSpider();

    spider.Initialize("www.chilkatsoft.com");

    String robotsText;
    robotsText = spider.fetchRobotsText();

    Log.i(TAG, robotsText);

  }

  static {
      System.loadLibrary("chilkat");

      // Note: If the incorrect library name is passed to System.loadLibrary,
      // then you will see the following error message at application startup:
      //"The application <your-application-name> has stopped unexpectedly. Please try again."
  }
}