Examples of Crawler

ca.forklabs.javaxpcom.Crawler
Class {@code Crawler} provides a skeleton to build web crawlers usingMozilla and SWT.
This crawler gives access to a {@link Browser} object inside a simple{@link Shell} that can be interacted with to navigate the web and access theunderlying {@link nsIDOMDocument}. @author Daniel Léonard @version $Revision$
com.crawl.control.Crawler
SINGLETON: The main crawler method to extract the oinformation from the Craiglists web page. @author Team Kappa
com.crawljax.core.Crawler
crawler.Crawler
net.sf.regain.crawler.Crawler
Durchsucht alle konfigurierten Startseiten nach URLs. Die gefundenen Seiten werden je nach Einstellung nur geladen, in den Suchindex aufgenommen oder wiederum nach URLs durchsucht.
für jede URL wird Anhand der Schwarzen und der Weißen Liste entschieden, ob sie ignoriert oder bearbeitet wird. Wenn loadUnparsedUrls auf false gesetzt wurde, dann werden auch URLs ignoriert, die weder durchsucht noch indiziert werden. @author Til Schneider, www.murfman.de
org.apache.cocoon.bean.helpers.Crawler
A simple Cocoon crawler @author Upayavira @version CVS $Id: Crawler.java,v 1.4 2004/03/05 13:02:45 bdelacretaz Exp $
placeholder.Crawler
Represents the crawler. @author Heiko Bornholdt
se.fidde.detective.crawlers.Crawler
websphinx.Crawler
Web crawler.
To write a crawler, extend this class and override shouldVisit () and visit() to create your own crawler.
To use a crawler:
1. Initialize the crawler by calling setRoot() (or one of its variants) and setting other crawl parameters.
2. Register any classifiers you need with addClassifier().
3. Connect event listeners to monitor the crawler, such as websphinx.EventLog, websphinx.workbench.WebGraph, or websphinx.workbench.Statistics.
4. Call run() to start the crawler.
A running crawler consists of a priority queue of Links waiting to be visited and a set of threads retrieving pages in parallel. When a page is downloaded, it is processed as follows:
1. classify(): The page is passed to the classify() method of every registered classifier, in increasing order of their priority values. Classifiers typically attach informative labels to the page and its links, such as "homepage" or "root page".
2. visit(): The page is passed to the crawler's visit() method for user-defined processing.
3. expand(): The page is passed to the crawler's expand() method to be expanded. The default implementation tests every unvisited hyperlink on the page with shouldVisit(), and puts each link approved by shouldVisit() into the crawling queue.
By default, when expanding the links of a page, the crawler only considers hyperlinks (not applets or inline images, for instance) that point to Web pages (not mailto: links, for instance). If you want shouldVisit() to test every link on the page, use setLinkType(Crawler.ALL_LINKS).

Examples of com.crawl.control.Crawler

    @Test
    public void test_MatchPatternCompareLength() {
        logger.debug("Test MatchPattern... compare length");
        
        // 1. Create the crawler object
        Crawler aCrawl = Crawler.getInstance();


        String aUrl=aCrawl.createUrl(
                CraigslistCategoryEnum.FOR_SALE__COMPUTER,
                CraigslistAreasEnum.PENINSULA, 
                "Apple");
               
        Collection<CrawlResultPackage> aCrawlResultColl=aCrawl.crawlWebPages(aUrl, 100);


        String aString=aCrawl.getMatchPattern();
        
        logger.info("MatchPatter="+aString+" length="+aString.length());
        
        assertTrue(aString.length()==43);
    }

View Full Code Here

Examples of com.crawl.control.Crawler

    @Test
    public void test_url1() {
        logger.debug("Test url 1");
        
        // 1. Create the crawler object
        Crawler aCrawl = Crawler.getInstance();


        String aUrl=aCrawl.createUrl(
                CraigslistCategoryEnum.FOR_SALE__COMPUTER,
                CraigslistAreasEnum.PENINSULA, 
                "Apple");
               
        Collection<CrawlResultPackage> aCrawlResultColl=aCrawl.crawlWebPages(aUrl, 100);
        
        logger.info("URL="+aUrl+" length="+aUrl.length());
        
        assertTrue(aUrl != null);
    }

View Full Code Here

Examples of com.crawl.control.Crawler

    @Test
    public void test_url2() {
        logger.debug("Test url 2...");
        
        // 1. Create the crawler object
        Crawler aCrawl = Crawler.getInstance();


        String aUrl=aCrawl.createUrl(
                CraigslistCategoryEnum.FOR_SALE__COMPUTER,
                CraigslistAreasEnum.PENINSULA, 
                "Apple");
               
        Collection<CrawlResultPackage> aCrawlResultColl=aCrawl.crawlWebPages(aUrl, 100);
        
        logger.info("URL="+aUrl+" length="+aUrl.length());
        
        assertTrue(aUrl.compareTo("http://sfbay.craigslist.org/search/sya/pen?query=Apple&maxAsk=100000&sort=pricedsc&srchType=A&s=") ==0);    
    }

View Full Code Here

Examples of com.crawl.control.Crawler

    @Test
    public void test_url3() {
        logger.debug("Test url 3...");
        
        // 1. Create the crawler object
        Crawler aCrawl = Crawler.getInstance();


        String aUrl=aCrawl.createUrl(
                CraigslistCategoryEnum.FOR_SALE__COMPUTER,
                CraigslistAreasEnum.PENINSULA, 
                "Apple");
               
        Collection<CrawlResultPackage> aCrawlResultColl=aCrawl.crawlWebPages(aUrl, 100);
                
        logger.debug("URL="+aUrl+" length="+aUrl.length());
        
        assertTrue(aUrl.length()==96);    
    }

View Full Code Here

Examples of com.crawljax.core.Crawler

    action = candidateActions.pollFirst();
    if (action != null) {
      workInProgressCandidateActions.put(requestingCrawler, action);
      return action;
    } else {
      Crawler c = registeredCrawlers.pollFirst();
      if (c == null) {
        return null;
      }
      do {
        if (manager.removeWorkFromQueue(c)) {

View Full Code Here

Examples of com.crawljax.core.Crawler

    action = candidateActions.pollFirst();
    if (action != null) {
      workInProgressCandidateActions.put(requestingCrawler, action);
      return action;
    } else {
      Crawler c = registeredCrawlers.pollFirst();
      if (c == null) {
        return null;
      }
      do {
        if (manager.removeWorkFromQueue(c)) {

View Full Code Here

Examples of com.crawljax.core.Crawler

    action = candidateActions.pollFirst();
    if (action != null) {
      workInProgressCandidateActions.put(requestingCrawler, action);
      return action;
    } else {
      Crawler c = registeredCrawlers.pollFirst();
      if (c == null) {
        return null;
      }
      do {
        if (manager.removeWorkFromQueue(c)) {

View Full Code Here

Examples of crawler.Crawler

     */
    public Crawler getCrawler(String crawlerId)
    {
        for (int i = 0; i < crawlers.size(); i++)
        {
            Crawler crawler = crawlers.get(i);
            String id = crawler.getCrawlerId();
            if (crawlerId.equals(id))
            {
                Crawler newCrawler = componentContext.getService(Crawler.class, "crawlers");
                return newCrawler;
            }
        }
        throw new RuntimeException("No crawler with id " + crawlerId + " found");
    }

View Full Code Here

Examples of net.sf.regain.crawler.Crawler

   * @param shouldPause Whether the crawler should pause.
   */
  public void setShouldPause(boolean shouldPause) {
    // NOTE: We get a local copy of the crawler for the case that is should
    //       change in the meantime
    Crawler crawler = mCrawler;
    if (crawler != null) {
      crawler.setShouldPause(shouldPause);
      TrayIconHandler.getInstance().setIndexUpdateRunning(! shouldPause);
    }
  }

View Full Code Here

Examples of net.sf.regain.crawler.Crawler


        // Create and run the crawler
        TrayIconHandler.getInstance().setIndexUpdateRunning(true);
        try {
          mLog.info("Starting index update on " + new Date());
          mCrawler = new Crawler(config, authProps);
          mCrawler.run(true, false, null);
        }
        catch (RegainException exc) {
          mLog.error("Updating the index failed", exc);
        }

View Full Code Here

0 1 2 3

TOP

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.