Printable Version of this PageHome PageRecent ChangesSearchSign In

Web Page Gatherer

Gather data: from specific web pages

The web page gatherer lets you specify a specific list of web pages from which you wish to gather data. This is useful when you do not want to start at a single web page and crawl all the links, but instead have a specific list of web pages in mind. For example, the My Webcams channel and the Atlanta Traffic channel use this approach. In the first example, there is no unifying web site that lists all of these webcams. Instead, each of their web sites is added to the list. In the Traffic example, each of the URLs for each resource on the web site is known, so they are added directly into the gatherer.

Options

OptionDescription
URLs A list of each of the URLs from which to gather data. Each URL can use a different scraper.


Advanced Options

You may sometimes need to change the advanced options.

OptionDescription
Filter data Specifies what filters to apply to the data scraped off of each web page.


Last modified 14 May 2007 at 3:33 pm by eaganj