Examples of CrawlableDataset

thredds.crawlabledataset.CrawlableDataset
CrawlableDataset represents an abstract dataset that is part of a hierarchical dataset collection. Parent and child datasets can be accessed allowing the collection to be crawled.

The CrawlableDataset interface is a generalization (and simplification) of the java.io.File class. A CrawlableDataset path is made up of ONE or more path segments each seperated by a slash ("/"). The path may start or end with a slash ("/").

Implementation Notes:
1. The thredds.crawlabledataset.CrawlableDatasetFactory requires each CrawlableDataset implementation to define a public constructor with one String argument and one Object argument. The String argument is the path for the CrawlableDataset being constructed, the Object argument is a configuration object.
2. The thredds.cataloggen.CollectionLevelScanner framework does not support building a catalog for the collection based at the CrawlableDataset path "/". So, do not implement your CrawlableDataset so that the path "/" is allowed, or at least so that it is not likely to be used as the root of a dataset collection (e.g., CrawlableDatasetFile is probably safe because the root directory ("/") should never be used as the base of a data collection. If the backend data source on top of which you are implementing CrawlabeDataset uses "/" as its root, you can simply prepend a string (e.g., "myDataCollection" or "root") to the backend path for the CrawlableDataset view of the path.
@author edavis @see thredds.cataloggen.CollectionLevelScanner CollectionLevelScanner uses CrawlableDatasets to scan a dataset collection and create a THREDDS catalog. @since May 3, 2005 20:18:59 -0600

Examples of thredds.crawlabledataset.CrawlableDataset

    String resultResourceName = resourcePath + "/" + resource_namedForDirWithNotEmptyServiceBase_result;
    File expectedCatalogDocFile = new File( resultResourceName );


    String collectionPath = "src/test/data/thredds/cataloggen/testData/modelNotFlat";
    String catalogPath = "src/test/data/thredds/cataloggen/testData/modelNotFlat/eta_211";
    CrawlableDataset collCrDs;
    CrawlableDataset catCrDs;
    try
    {
      collCrDs = CrawlableDatasetFactory.createCrawlableDataset( collectionPath, null, null );
      catCrDs = CrawlableDatasetFactory.createCrawlableDataset( catalogPath, null, null );
    }

View Full Code Here

Examples of thredds.crawlabledataset.CrawlableDataset

    return( this.getDirCatalog( directory, filterPattern, sortInIncreasingOrder, null, addDatasetSize, null, null, null) );
  }


  public InvCatalog getDirCatalog( File directory, String filterPattern, boolean sortInIncreasingOrder, String addIdBase, boolean addDatasetSize, String dsNameMatchPattern, String startTimeSubstitutionPattern, String duration )
  {
    CrawlableDataset catalogCrDs;
    try
    {
      catalogCrDs = CrawlableDatasetFactory.createCrawlableDataset( directory.getAbsolutePath(), null, null );
    }
    catch ( IOException e )
    {
      // @todo Should throw an IOException!
      throw new IllegalArgumentException( "IOException while creating dataset: " + e.getMessage() );
    }
    catch ( ClassNotFoundException e )
    {
      throw new IllegalArgumentException( "Did not find class: " + e.getMessage() );
    }
    catch ( NoSuchMethodException e )
    {
      throw new IllegalArgumentException( "Required constructor not found in class: " + e.getMessage() );
    }
    catch ( IllegalAccessException e )
    {
      throw new IllegalArgumentException( "Did not have necessary access to class: " + e.getMessage() );
    }
    catch ( InvocationTargetException e )
    {
      throw new IllegalArgumentException( "Could not invoke required method in class: " + e.getMessage() );
    }
    catch ( InstantiationException e )
    {
      throw new IllegalArgumentException( "Could not instatiate class: " + e.getMessage() );
    }
    if ( ! catalogCrDs.isCollection() )
      throw new IllegalArgumentException( "catalog directory is not a directory <" + serviceBaseUrlDir.getAbsolutePath() + ">." );


    return getDirCatalog( catalogCrDs, filterPattern, sortInIncreasingOrder, addIdBase, addDatasetSize, dsNameMatchPattern, startTimeSubstitutionPattern, duration );
  }

View Full Code Here

Examples of thredds.crawlabledataset.CrawlableDataset

    String remainingPath = path.substring( ancestorCrDs.getPath().length() );
    if ( remainingPath.startsWith( "/" ) )
      remainingPath = remainingPath.substring( 1 );


    String[] pathSegments = remainingPath.split( "/" );
    CrawlableDataset curCrDs = ancestorCrDs;
    for ( int i = 0; i < pathSegments.length; i++ )
    {
      curCrDs = curCrDs.getDescendant( pathSegments[i]);
      if ( filter != null )
        if ( ! filter.accept( curCrDs ) )
          return null;
    }
    // Only check complete path for existence since speed of check depends on implementation.
    if ( ! curCrDs.exists() )
      return null;
    return curCrDs;
  }

View Full Code Here

Examples of thredds.crawlabledataset.CrawlableDataset

    }


    File catWriteDir = new File( catWriteDirPath );


    File collectionFile = new File( collectionPath );
    CrawlableDataset collectionCrDs = new CrawlableDatasetFile( collectionFile );
    InvService service = new InvService( "myServer", "File", collectionCrDs.getPath() + "/", null, null);
    CrawlableDatasetFilter filter = null;
    CrawlableDataset topCatCrDs = collectionCrDs.getDescendant( startPath );


    CatGenAndWrite cgaw = new CatGenAndWrite( "DATA", "My data", "", service,
                                              collectionCrDs, topCatCrDs, filter, null, catWriteDir );


    try
    {
      cgaw.genCatAndSubCats( topCatCrDs );
    }
    catch ( IOException e )
    {
      log.error( "I/O error generating and writing catalogs at and under \"" + topCatCrDs.getPath() + "\": " + e.getMessage());
      return;
    }


  }

View Full Code Here

Examples of thredds.crawlabledataset.CrawlableDataset

    }


    File catWriteDir = new File( catWriteDirPath );


    File collectionFile = new File( collectionPath );
    CrawlableDataset collectionCrDs = new CrawlableDatasetFile( collectionFile );
    InvService service = new InvService( "myServer", "OPENDAP", "/thredds/dodsC/", null, null );
    CrawlableDatasetFilter filter = null;
    CrawlableDataset topCatCrDs = collectionCrDs.getDescendant( startPath );


    CatGenAndWrite cgaw = new CatGenAndWrite( "DATA", "My data", "mlode", service,
                                              collectionCrDs, topCatCrDs, filter, null, catWriteDir );


    try
    {
      cgaw.genCatAndSubCats( topCatCrDs );
    }
    catch ( IOException e )
    {
      log.error( "I/O error generating and writing catalogs at and under \"" + topCatCrDs.getPath() + "\": " + e.getMessage() );
      return;
    }
  }

View Full Code Here

Examples of thredds.crawlabledataset.CrawlableDataset


    // Find child datasets that are collections and generate catalogs for those as well.
    List collectionChildren = catCrDs.listDatasets( collectionOnlyFilter);
    for ( Iterator it = collectionChildren.iterator(); it.hasNext(); )
    {
      CrawlableDataset curCrDs = (CrawlableDataset) it.next();


      genCatAndSubCats( curCrDs );
    }


  }

View Full Code Here

Examples of thredds.crawlabledataset.CrawlableDataset

  public CrawlableDataset getParentDataset()
  {
    if ( ! this.delegate.exists() )
      return null;


    CrawlableDataset possibleParent = this.delegate.getParentDataset();
    if ( possibleParent == null )
      return null;


    if ( crDsTree.containsKey( possibleParent.getPath() ))
      return new MockCrawlableDatasetTree( possibleParent.getPath(), this.crDsTree );
    // ToDo
    return this.delegate.getParentDataset();
  }

View Full Code Here

Examples of thredds.crawlabledataset.CrawlableDataset


  private void checkRegExpMatch( String regExpString, String dsPath )
  {
    RegExpMatchOnNameFilter me = new RegExpMatchOnNameFilter( regExpString );


    CrawlableDataset ds = null;
    try
    {
      ds = CrawlableDatasetFactory.createCrawlableDataset( dsPath, null, null );
    }
    catch ( Exception e )
    {
      assertTrue( "Failed to create CrawlableDataset <" + dsPath + ">: " + e.getMessage(),
                  false );
    }
    assertTrue( "Failed to construct RegExpMatchOnNameFilter <" + regExpString + ">.",
                me != null );


    assertTrue( "RegExp string <" + regExpString + "> did not match dataset <" + ds.getName() + ">.",
                me.accept( ds) );
  }

View Full Code Here

Examples of thredds.crawlabledataset.CrawlableDataset

            LogicalFilterComposer.getAndFilter( includeNcFilter,
            LogicalFilterComposer.getNotFilter( lastModAtLeast4MinPastFilter ) );


    CrawlableDatasetFilter oldGribOrNewNcFilter = LogicalFilterComposer.getOrFilter( oldGribFilter, newNcFilter);


    CrawlableDataset tmpDirCrDs = new CrawlableDatasetFile( tmpDir);
    List crDsList = null;
    try
    {
      crDsList = tmpDirCrDs.listDatasets();
    }
    catch ( IOException e )
    {
      assertTrue( "I/O problem getting contained dataset list.",
                  false );
      deleteFiles();
      return;
    }
    for ( Iterator it = crDsList.iterator(); it.hasNext(); )
    {
      CrawlableDataset curCrDs = (CrawlableDataset) it.next();
      if ( oldGribOrNewNcFilter.accept( curCrDs) )
      {
        if ( ! curCrDs.getName().equals( "old.grib1")
             && ! curCrDs.getName().equals( "new.nc"))
        {
          assertTrue( "Matched wrong file <" + curCrDs.getPath() + ">.",
                      false);
          deleteFiles();
          return;
        }
      }

View Full Code Here

Examples of thredds.crawlabledataset.CrawlableDataset


  private void checkWildcardMatch( String wildcardString, String dsPath )
  {
    WildcardMatchOnNameFilter me = new WildcardMatchOnNameFilter( wildcardString );


    CrawlableDataset ds = null;
    try
    {
      ds = CrawlableDatasetFactory.createCrawlableDataset( dsPath, null, null );
    }
    catch ( Exception e )
    {
      assertTrue( "Failed to create CrawlableDataset <" + dsPath + ">: " + e.getMessage(),
                  false );
    }
    assertTrue( "Failed to construct WildcardMatchOnNameFilter <" + wildcardString + ">.",
                me != null );


    assertTrue( "Wildcard string <" + wildcardString + "> did not match dataset <" + ds.getName() + ">.",
                me.accept( ds) );
  }

View Full Code Here

0 1 2

TOP

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.