List of addCacheFile() Examples

Examples of addCacheFile()

org.apache.hadoop.mapreduce.Job.addCacheFile()
Add a file to be localized @param uri The uri of the cache to be localized

Examples of org.apache.hadoop.mapreduce.Job.addCacheFile()

      Path partitionFile = new Path(outputDir, 
                                    TeraInputFormat.PARTITION_FILENAME);
      URI partitionUri = new URI(partitionFile.toString() +
                                 "#" + TeraInputFormat.PARTITION_FILENAME);
      TeraInputFormat.writePartitionFile(job, partitionFile);
      job.addCacheFile(partitionUri);
      job.createSymlink();    
      long end = System.currentTimeMillis();
      System.out.println("Spent " + (end - start) + "ms computing partitions.");
      job.setPartitionerClass(TotalOrderPartitioner.class);
    }

View Full Code Here

Examples of org.apache.hadoop.mapreduce.Job.addCacheFile()

    conf.set(JobContext.MAP_DEBUG_SCRIPT, "/bin/echo");
    conf.set(JobContext.REDUCE_DEBUG_SCRIPT, "/bin/echo");
    Job j = MapReduceTestUtil.createJob(conf, new Path(TEST_DIR, "in"),
        new Path(TEST_DIR, "out"), 0, 0);
    // Add the local filed created to the cache files of the job
    j.addCacheFile(new URI(CACHE_FILE_PATH));
    j.setMaxMapAttempts(1);
    j.setMaxReduceAttempts(1);
    // Submit the job and return immediately.
    // Job submit now takes care setting the last
    // modified time of the cache file.

View Full Code Here

Examples of org.apache.hadoop.mapreduce.Job.addCacheFile()

    Job job = Job.getInstance(conf);
    job.setMapperClass(DistributedCacheChecker.class);
    job.setOutputFormatClass(NullOutputFormat.class);
    FileInputFormat.setInputPaths(job, first);
    // Creates the Job Configuration
    job.addCacheFile(
      new URI(first.toUri().toString() + "#distributed.first.symlink"));
    job.addFileToClassPath(second);
    job.addArchiveToClassPath(third);
    job.addCacheArchive(fourth.toUri());
    job.createSymlink();

View Full Code Here

Examples of org.apache.hadoop.mapreduce.Job.addCacheFile()

    File workDir = new File(new Path(TEST_ROOT_DIR, "workdir").toString());


    // Configures a job with a regular file
    Job job1 = Job.getInstance(cluster, conf);
    job1.setUser(userName);
    job1.addCacheFile(secondCacheFile.toUri());
    Configuration conf1 = job1.getConfiguration();
    TrackerDistributedCacheManager.determineTimestamps(conf1);
    TrackerDistributedCacheManager.determineCacheVisibilities(conf1);


    // Task localizing for first job

View Full Code Here

Examples of org.apache.hadoop.mapreduce.Job.addCacheFile()

    
    // Configures another job with three regular files.
    Job job2 = Job.getInstance(cluster, conf);
    job2.setUser(userName);
    // add a file that would get failed to localize
    job2.addCacheFile(firstCacheFile.toUri());
    // add a file that is already localized by different job
    job2.addCacheFile(secondCacheFile.toUri());
    // add a file that is never localized
    job2.addCacheFile(thirdCacheFile.toUri());
    Configuration conf2 = job2.getConfiguration();

View Full Code Here

Examples of org.apache.hadoop.mapreduce.Job.addCacheFile()

    Job job2 = Job.getInstance(cluster, conf);
    job2.setUser(userName);
    // add a file that would get failed to localize
    job2.addCacheFile(firstCacheFile.toUri());
    // add a file that is already localized by different job
    job2.addCacheFile(secondCacheFile.toUri());
    // add a file that is never localized
    job2.addCacheFile(thirdCacheFile.toUri());
    Configuration conf2 = job2.getConfiguration();
    TrackerDistributedCacheManager.determineTimestamps(conf2);
    TrackerDistributedCacheManager.determineCacheVisibilities(conf2);

View Full Code Here

Examples of org.apache.hadoop.mapreduce.Job.addCacheFile()

    // add a file that would get failed to localize
    job2.addCacheFile(firstCacheFile.toUri());
    // add a file that is already localized by different job
    job2.addCacheFile(secondCacheFile.toUri());
    // add a file that is never localized
    job2.addCacheFile(thirdCacheFile.toUri());
    Configuration conf2 = job2.getConfiguration();
    TrackerDistributedCacheManager.determineTimestamps(conf2);
    TrackerDistributedCacheManager.determineCacheVisibilities(conf2);


    // Task localizing for second job

View Full Code Here

Examples of org.apache.hadoop.mapreduce.Job.addCacheFile()

      createPrivateTempFile(cacheFile);
    }
    
    Job job1 = Job.getInstance(cluster, conf);
    job1.setUser(userName);
    job1.addCacheFile(cacheFile.toUri());
    Configuration conf1 = job1.getConfiguration();
    TrackerDistributedCacheManager.determineTimestamps(conf1);
    TrackerDistributedCacheManager.determineCacheVisibilities(conf1);


    // Task localizing for job

View Full Code Here

Examples of org.apache.hadoop.mapreduce.Job.addCacheFile()

  }
  
  @Test
  public void testDetermineTimestamps() throws IOException {
    Job job = Job.getInstance(conf);
    job.addCacheFile(firstCacheFile.toUri());
    job.addCacheFile(secondCacheFile.toUri());
    Configuration jobConf = job.getConfiguration();
    
    Map<URI, FileStatus> statCache = new HashMap<URI, FileStatus>();
    ClientDistributedCacheManager.determineTimestamps(jobConf, statCache);

View Full Code Here

Examples of org.apache.hadoop.mapreduce.Job.addCacheFile()

  
  @Test
  public void testDetermineTimestamps() throws IOException {
    Job job = Job.getInstance(conf);
    job.addCacheFile(firstCacheFile.toUri());
    job.addCacheFile(secondCacheFile.toUri());
    Configuration jobConf = job.getConfiguration();
    
    Map<URI, FileStatus> statCache = new HashMap<URI, FileStatus>();
    ClientDistributedCacheManager.determineTimestamps(jobConf, statCache);

View Full Code Here

0 1 2 3 4

TOP

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.