Lua

Must-Match Patterns

See more Spider Examples

You may restrict the spider to only follow links that match any one of a set of "must-match" wildcard patterns. The AddMustMatchPattern can be called repeatedly to add must-match patterns. Lua

local chilkat = require("chilkat")

local spider = chilkat.Spider.new()

-- --------------------------------------------------------------------
-- Note: The URLs in this example are no longer valid.
-- You should replace the URLs with URLs from a site of your
-- own choosing -- preferably your own site if testing.
-- (Google's Directory no longer exists.)
-- --------------------------------------------------------------------

-- First, we'll get the outbound links for a page in the
-- Google directory.  Then we'll add some must-match
-- and then re-fetch, to see it work...

spider:initialize("directory.google.com")
spider:add_unspidered("http://directory.google.com/Top/Recreation/Outdoors/Hiking/Backpacking/")

local success = spider:crawl_next()

-- Display the outbound links
local url = ""
for i = 0, spider.num_outbound_links - 1 do
    print(spider:get_outbound_link(i))
end

-- The output:
-- http://www.backpacker.com
-- http://www.cmc.org
-- http://www.backpacking.net
-- http://www.thebackpacker.com/
-- http://www.rei.com/online/store/LearnShareArticlesList?categoryId=Camping
-- http://www.trailspace.com/
-- http://www.catskillhikes.com/
-- http://gorp.away.com/gorp/location/asia/nepal/favpicks.htm
-- http://www.backpackinglight.com/cgi-bin/backpackinglight/index.html
-- http://www.yetizone.com/
-- http://www.backpackingfun.com
-- http://www.freezerbagcooking.com/
-- http://www.spadout.com/backpacking/
-- http://sierrabackpacker.com
-- http://www.abovecalifornia.com/
-- http://www.personal.psu.edu/faculty/r/p/rpc1/bbb/
-- http://www.thebackpackersguide.com
-- http://www.journeywest.com/WB/index.html
-- http://www.johann-sandra.com/backpackdir.htm
-- http://www.geocities.com/amytys/
-- http://www.cloudwalkersbasecamp.com
-- http://www.netbackpacking.com
-- http://members.tripod.com/~stooges/
-- http://www.thebackpackingsite.com
-- http://www.thruhikers.com/
-- http://www.redcompservices.com/AT/
-- http://members.aol.com/CMorHiker/backpack
-- http://mywebpages.comcast.net/midwestpacker/
-- http://www.midwesthiker.com/
-- http://www.WeBackpack.com
-- http://www.michiganhiker.com
-- http://www.host33.com/backpack/
-- http://www.wilderness-backpacking.com
-- http://www.thetravelmonkey.net
-- http://dmoz.org/cgi-bin/add.cgi?where=Recreation/Outdoors/Hiking/Backpacking
-- http://dmoz.org/about.html
-- http://dmoz.org/cgi-bin/apply.cgi?where=Recreation/Outdoors/Hiking/Backpacking
-- http://dmoz.org
-- http://dmoz.org/profiles/cdog.html
-- http://dmoz.org/profiles/justinwp.html

-- Do it again, but this time with avoid patterns.
spider:initialize("directory.google.com")
spider:add_unspidered("http://directory.google.com/Top/Recreation/Outdoors/Hiking/Backpacking/")

-- Add some must-match patterns:
spider:add_must_match_pattern("*.com/*")
spider:add_must_match_pattern("*.net/*")

-- Add some avoid-patterns:
spider:add_avoid_outbound_link_pattern("*.mypages.*")
spider:add_avoid_outbound_link_pattern("*.personal.*")
spider:add_avoid_outbound_link_pattern("*.comcast.*")
spider:add_avoid_outbound_link_pattern("*.aol.*")
spider:add_avoid_outbound_link_pattern("*~*")

success = spider:crawl_next()

print("-----------------------")

-- Display the outbound links
for i = 0, spider.num_outbound_links - 1 do
    print(spider:get_outbound_link(i))
end

-- Output:
-- http://www.thebackpacker.com/
-- http://www.rei.com/online/store/LearnShareArticlesList?categoryId=Camping
-- http://www.trailspace.com/
-- http://www.catskillhikes.com/
-- http://gorp.away.com/gorp/location/asia/nepal/favpicks.htm
-- http://www.backpackinglight.com/cgi-bin/backpackinglight/index.html
-- http://www.yetizone.com/
-- http://www.freezerbagcooking.com/
-- http://www.spadout.com/backpacking/
-- http://www.abovecalifornia.com/
-- http://www.journeywest.com/WB/index.html
-- http://www.johann-sandra.com/backpackdir.htm
-- http://www.geocities.com/amytys/
-- http://www.thruhikers.com/
-- http://www.redcompservices.com/AT/
-- http://www.midwesthiker.com/
-- http://www.host33.com/backpack/