List of buildListing() Examples

Examples of buildListing()

org.apache.hadoop.tools.CopyListing.buildListing()
Build listing function creates the input listing that distcp uses to perform the copy. The build listing is a sequence file that has relative path of a file in the key and the file status information of the source file in the value For instance if the source path is /tmp/data and the traversed path is /tmp/data/dir1/dir2/file1, then the sequence file would contain key: /dir1/dir2/file1 and value: FileStatus(/tmp/data/dir1/dir2/file1) File would also contain directory entries. Meaning, if /tmp/data/dir1/dir2/file1 is the only file under /tmp/data, the resulting sequence file would contain the following entries key: /dir1 and value: FileStatus(/tmp/data/dir1) key: /dir1/dir2 and value: FileStatus(/tmp/data/dir1/dir2) key: /dir1/dir2/file1 and value: FileStatus(/tmp/data/dir1/dir2/file1) Cases requiring special handling: If source path is a file (/tmp/file1), contents of the file will be as follows TARGET DOES NOT EXIST: Key-"", Value-FileStatus(/tmp/file1) TARGET IS FILE : Key-"", Value-FileStatus(/tmp/file1) TARGET IS DIR : Key-"/file1", Value-FileStatus(/tmp/file1) @param pathToListFile - Output file where the listing would be stored @param options - Input options to distcp @throws IOException - Exception if any
org.apache.hadoop.tools.GlobbedCopyListing.buildListing()
org.apache.hadoop.tools.distcp2.CopyListing.buildListing()
Build listing function creates the input listing that distcp uses to perform the copy. The build listing is a sequence file that has relative path of a file in the key and the file status information of the source file in the value For instance if the source path is /tmp/data and the traversed path is /tmp/data/dir1/dir2/file1, then the sequence file would contain key: /dir1/dir2/file1 and value: FileStatus(/tmp/data/dir1/dir2/file1) File would also contain directory entries. Meaning, if /tmp/data/dir1/dir2/file1 is the only file under /tmp/data, the resulting sequence file would contain the following entries key: /dir1 and value: FileStatus(/tmp/data/dir1) key: /dir1/dir2 and value: FileStatus(/tmp/data/dir1/dir2) key: /dir1/dir2/file1 and value: FileStatus(/tmp/data/dir1/dir2/file1) Cases requiring special handling: If source path is a file (/tmp/file1), contents of the file will be as follows TARGET DOES NOT EXIST: Key-"", Value-FileStatus(/tmp/file1) TARGET IS FILE : Key-"", Value-FileStatus(/tmp/file1) TARGET IS DIR : Key-"/file1", Value-FileStatus(/tmp/file1) @param pathToListFile - Output file where the listing would be stored @param options - Input options to distcp @throws IOException - Exception if any
org.apache.hadoop.tools.distcp2.GlobbedCopyListing.buildListing()
Build listing function creates the input listing that distcp uses to perform the copy. The build listing is a sequence file that has relative path of a file in the key and the file status information of the source file in the value For instance if the source path is /tmp/data and the traversed path is /tmp/data/dir1/dir2/file1, then the sequence file would contain key: /dir1/dir2/file1 and value: FileStatus(/tmp/data/dir1/dir2/file1) File would also contain directory entries. Meaning, if /tmp/data/dir1/dir2/file1 is the only file under /tmp/data, the resulting sequence file would contain the following entries key: /dir1 and value: FileStatus(/tmp/data/dir1) key: /dir1/dir2 and value: FileStatus(/tmp/data/dir1/dir2) key: /dir1/dir2/file1 and value: FileStatus(/tmp/data/dir1/dir2/file1) Cases requiring special handling: If source path is a file (/tmp/file1), contents of the file will be as follows TARGET DOES NOT EXIST: Key-"", Value-FileStatus(/tmp/file1) TARGET IS FILE : Key-"", Value-FileStatus(/tmp/file1) TARGET IS DIR : Key-"/file1", Value-FileStatus(/tmp/file1) @param pathToListFile - Output file where the listing would be stored @param options - Input options to distcp @throws IOException - Exception if any
org.teavm.model.util.ListingBuilder.buildListing()

Examples of org.apache.hadoop.tools.GlobbedCopyListing.buildListing()

      options.setDeleteMissing(true);
      options.appendToConf(conf);


      CopyListing listing = new GlobbedCopyListing(conf, CREDENTIALS);
      Path listingFile = new Path("/tmp1/" + String.valueOf(rand.nextLong()));
      listing.buildListing(listingFile, options);


      conf.set(DistCpConstants.CONF_LABEL_TARGET_WORK_PATH, targetBase);
      conf.set(DistCpConstants.CONF_LABEL_TARGET_FINAL_PATH, targetBase);


      committer.commitJob(jobContext);

View Full Code Here

Examples of org.apache.hadoop.tools.distcp2.CopyListing.buildListing()

      options.preserve(FileAttribute.PERMISSION);
      options.appendToConf(conf);


      CopyListing listing = new GlobbedCopyListing(conf, CREDENTIALS);
      Path listingFile = new Path("/tmp1/" + String.valueOf(rand.nextLong()));
      listing.buildListing(listingFile, options);


      conf.set(DistCpConstants.CONF_LABEL_TARGET_WORK_PATH, targetBase);


      committer.commitJob(jobContext);
      if (!checkDirectoryPermissions(fs, targetBase, sourcePerm)) {

View Full Code Here

Examples of org.apache.hadoop.tools.distcp2.CopyListing.buildListing()

      options.setDeleteMissing(true);
      options.appendToConf(conf);


      CopyListing listing = new GlobbedCopyListing(conf, CREDENTIALS);
      Path listingFile = new Path("/tmp1/" + String.valueOf(rand.nextLong()));
      listing.buildListing(listingFile, options);


      conf.set(DistCpConstants.CONF_LABEL_TARGET_WORK_PATH, targetBase);
      conf.set(DistCpConstants.CONF_LABEL_TARGET_FINAL_PATH, targetBase);


      committer.commitJob(jobContext);

View Full Code Here

Examples of org.apache.hadoop.tools.distcp2.CopyListing.buildListing()

      options.setDeleteMissing(true);
      options.appendToConf(conf);


      CopyListing listing = new GlobbedCopyListing(conf, CREDENTIALS);
      Path listingFile = new Path("/tmp1/" + String.valueOf(rand.nextLong()));
      listing.buildListing(listingFile, options);


      conf.set(DistCpConstants.CONF_LABEL_TARGET_WORK_PATH, targetBase);
      conf.set(DistCpConstants.CONF_LABEL_TARGET_FINAL_PATH, targetBase);


      committer.commitJob(jobContext);

View Full Code Here

Examples of org.apache.hadoop.tools.distcp2.CopyListing.buildListing()

    List<Path> targets = new ArrayList<Path>(1);
    Path targetFinalPath = new Path(conf.get(DistCpConstants.CONF_LABEL_TARGET_FINAL_PATH));
    targets.add(targetFinalPath);
    DistCpOptions options = new DistCpOptions(targets, new Path("/NONE"));


    target.buildListing(targetListing, options);
    Path sortedTargetListing = DistCpUtils.sortListing(clusterFS, conf, targetListing);
    long totalLen = clusterFS.getFileStatus(sortedTargetListing).getLen();


    SequenceFile.Reader sourceReader = new SequenceFile.Reader(
        clusterFS, sortedSourceListing, conf);

View Full Code Here

Examples of org.apache.hadoop.tools.distcp2.GlobbedCopyListing.buildListing()

      options.preserve(FileAttribute.PERMISSION);
      options.appendToConf(conf);


      CopyListing listing = new GlobbedCopyListing(conf, CREDENTIALS);
      Path listingFile = new Path("/tmp1/" + String.valueOf(rand.nextLong()));
      listing.buildListing(listingFile, options);


      conf.set(DistCpConstants.CONF_LABEL_TARGET_WORK_PATH, targetBase);


      committer.commitJob(jobContext);
      if (!checkDirectoryPermissions(fs, targetBase, sourcePerm)) {

View Full Code Here

Examples of org.apache.hadoop.tools.distcp2.GlobbedCopyListing.buildListing()

      options.setDeleteMissing(true);
      options.appendToConf(conf);


      CopyListing listing = new GlobbedCopyListing(conf, CREDENTIALS);
      Path listingFile = new Path("/tmp1/" + String.valueOf(rand.nextLong()));
      listing.buildListing(listingFile, options);


      conf.set(DistCpConstants.CONF_LABEL_TARGET_WORK_PATH, targetBase);
      conf.set(DistCpConstants.CONF_LABEL_TARGET_FINAL_PATH, targetBase);


      committer.commitJob(jobContext);

View Full Code Here

Examples of org.apache.hadoop.tools.distcp2.GlobbedCopyListing.buildListing()

      options.setDeleteMissing(true);
      options.appendToConf(conf);


      CopyListing listing = new GlobbedCopyListing(conf, CREDENTIALS);
      Path listingFile = new Path("/tmp1/" + String.valueOf(rand.nextLong()));
      listing.buildListing(listingFile, options);


      conf.set(DistCpConstants.CONF_LABEL_TARGET_WORK_PATH, targetBase);
      conf.set(DistCpConstants.CONF_LABEL_TARGET_FINAL_PATH, targetBase);


      committer.commitJob(jobContext);

View Full Code Here

Examples of org.apache.hadoop.tools.distcp2.GlobbedCopyListing.buildListing()

    List<Path> targets = new ArrayList<Path>(1);
    Path targetFinalPath = new Path(conf.get(DistCpConstants.CONF_LABEL_TARGET_FINAL_PATH));
    targets.add(targetFinalPath);
    DistCpOptions options = new DistCpOptions(targets, new Path("/NONE"));


    target.buildListing(targetListing, options);
    Path sortedTargetListing = DistCpUtils.sortListing(clusterFS, conf, targetListing);
    long totalLen = clusterFS.getFileStatus(sortedTargetListing).getLen();


    SequenceFile.Reader sourceReader = new SequenceFile.Reader(
        clusterFS, sortedSourceListing, conf);

View Full Code Here

Examples of org.teavm.model.util.ListingBuilder.buildListing()

        }
        writer.println(")");
        Program program = method.getProgram();
        if (program != null && program.basicBlockCount() > 0) {
            ListingBuilder builder = new ListingBuilder();
            writer.print(builder.buildListing(program, "        "));
            writer.print("        Register allocation:");
            for (int i = 0; i < program.variableCount(); ++i) {
                writer.print(i + ":" + program.variableAt(i).getRegister() + " ");
            }
            writer.println();

View Full Code Here

0 1 2 3

TOP

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.