List of addCacheFile() Examples

Examples of addCacheFile()

org.apache.hadoop.mapreduce.Job.addCacheFile()
Add a file to be localized @param uri The uri of the cache to be localized

Examples of org.apache.hadoop.mapreduce.Job.addCacheFile()

        TeraInputFormat.writePartitionFile(job, partitionFile);
      } catch (Throwable e) {
        LOG.error(e.getMessage());
        return -1;
      }
      job.addCacheFile(partitionUri);
      job.createSymlink();    
      long end = System.currentTimeMillis();
      System.out.println("Spent " + (end - start) + "ms computing partitions.");
      job.setPartitionerClass(TotalOrderPartitioner.class);
    }

View Full Code Here

Examples of org.apache.hadoop.mapreduce.Job.addCacheFile()

    Job job = Job.getInstance(conf);
    job.setMapperClass(DistributedCacheChecker.class);
    job.setOutputFormatClass(NullOutputFormat.class);
    FileInputFormat.setInputPaths(job, first);
    // Creates the Job Configuration
    job.addCacheFile(
      new URI(first.toUri().toString() + "#distributed.first.symlink"));
    job.addFileToClassPath(second);
    job.addArchiveToClassPath(third);
    job.addCacheArchive(fourth.toUri());
    job.setMaxMapAttempts(1); // speed up failures

View Full Code Here

Examples of org.apache.hadoop.mapreduce.Job.addCacheFile()

        TeraInputFormat.writePartitionFile(job, partitionFile);
      } catch (Throwable e) {
        LOG.error(e.getMessage());
        return -1;
      }
      job.addCacheFile(partitionUri);  
      long end = System.currentTimeMillis();
      System.out.println("Spent " + (end - start) + "ms computing partitions.");
      job.setPartitionerClass(TotalOrderPartitioner.class);
    }

View Full Code Here

Examples of org.apache.hadoop.mapreduce.Job.addCacheFile()

    File workDir = new File(new Path(TEST_ROOT_DIR, "workdir").toString());


    // Configures a job with a regular file
    Job job1 = Job.getInstance(conf);
    job1.setUser(userName);
    job1.addCacheFile(secondCacheFile.toUri());
    Configuration conf1 = job1.getConfiguration();
    ClientDistributedCacheManager.determineTimestamps(conf1);
    ClientDistributedCacheManager.determineCacheVisibilities(conf1);


    // Task localizing for first job

View Full Code Here

Examples of org.apache.hadoop.mapreduce.Job.addCacheFile()

    
    // Configures another job with three regular files.
    Job job2 = Job.getInstance(conf);
    job2.setUser(userName);
    // add a file that would get failed to localize
    job2.addCacheFile(firstCacheFile.toUri());
    // add a file that is already localized by different job
    job2.addCacheFile(secondCacheFile.toUri());
    // add a file that is never localized
    job2.addCacheFile(thirdCacheFile.toUri());
    Configuration conf2 = job2.getConfiguration();

View Full Code Here

Examples of org.apache.hadoop.mapreduce.Job.addCacheFile()

    Job job2 = Job.getInstance(conf);
    job2.setUser(userName);
    // add a file that would get failed to localize
    job2.addCacheFile(firstCacheFile.toUri());
    // add a file that is already localized by different job
    job2.addCacheFile(secondCacheFile.toUri());
    // add a file that is never localized
    job2.addCacheFile(thirdCacheFile.toUri());
    Configuration conf2 = job2.getConfiguration();
    ClientDistributedCacheManager.determineTimestamps(conf2);
    ClientDistributedCacheManager.determineCacheVisibilities(conf2);

View Full Code Here

Examples of org.apache.hadoop.mapreduce.Job.addCacheFile()

    // add a file that would get failed to localize
    job2.addCacheFile(firstCacheFile.toUri());
    // add a file that is already localized by different job
    job2.addCacheFile(secondCacheFile.toUri());
    // add a file that is never localized
    job2.addCacheFile(thirdCacheFile.toUri());
    Configuration conf2 = job2.getConfiguration();
    ClientDistributedCacheManager.determineTimestamps(conf2);
    ClientDistributedCacheManager.determineCacheVisibilities(conf2);


    // Task localizing for second job

View Full Code Here

Examples of org.apache.hadoop.mapreduce.Job.addCacheFile()

      createPrivateTempFile(cacheFile);
    }
    
    Job job1 = Job.getInstance(conf);
    job1.setUser(userName);
    job1.addCacheFile(cacheFile.toUri());
    Configuration conf1 = job1.getConfiguration();
    ClientDistributedCacheManager.determineTimestamps(conf1);
    ClientDistributedCacheManager.determineCacheVisibilities(conf1);


    // Task localizing for job

View Full Code Here

Examples of org.apache.hadoop.mapreduce.Job.addCacheFile()

    job.setMapperClass(DistributedCacheChecker.class);
    job.setOutputFormatClass(NullOutputFormat.class);


    FileInputFormat.setInputPaths(job, first);
    // Creates the Job Configuration
    job.addCacheFile(
        new URI(first.toUri().toString() + "#distributed.first.symlink"));
    job.addFileToClassPath(second);
    job.addFileToClassPath(APP_JAR); // The AppMaster jar itself.
    job.addArchiveToClassPath(third);
    job.addCacheArchive(fourth.toUri());

View Full Code Here

Examples of org.apache.hadoop.mapreduce.Job.addCacheFile()

    Job job = Job.getInstance(conf);
    job.setMapperClass(DistributedCacheChecker.class);
    job.setOutputFormatClass(NullOutputFormat.class);
    FileInputFormat.setInputPaths(job, first);
    // Creates the Job Configuration
    job.addCacheFile(
      new URI(first.toUri().toString() + "#distributed.first.symlink"));
    job.addFileToClassPath(second);
    job.addArchiveToClassPath(third);
    job.addCacheArchive(fourth.toUri());

View Full Code Here

0 1 2 3 4

TOP

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.