Examples of org.apache.crunch.hadoop.mapreduce.lib.jobcontrol.CrunchControlledJob

Package org.apache.crunch.hadoop.mapreduce.lib.jobcontrol

Examples of org.apache.crunch.hadoop.mapreduce.lib.jobcontrol.CrunchControlledJob

org.apache.crunch.hadoop.mapreduce.lib.jobcontrol.CrunchControlledJob
This class encapsulates a MapReduce job and its dependency. It monitors the states of the depending jobs and updates the state of this job. A job starts in the WAITING state. If it does not have any depending jobs, or all of the depending jobs are in SUCCESS state, then the job state will become READY. If any depending jobs fail, the job will fail too. When in READY state, the job can be submitted to Hadoop for execution, with the state changing into RUNNING state. From RUNNING state, the job can get into SUCCEEDED or FAILED state, depending the status of the job execution.

      }
      job.setInputFormatClass(CrunchInputFormat.class);
    }
    job.setJobName(createJobName(conf, pipeline.getName(), inputNodes, reduceNode, numOfJobs));


    return new CrunchControlledJob(
        jobID,
        job,
        new CrunchJobHooks.PrepareHook(job),
        new CrunchJobHooks.CompletionHook(job, outputPath, outputHandler.getMultiPaths(), group == null));
  }

View Full Code Here

      }
      job.setInputFormatClass(CrunchInputFormat.class);
    }
    job.setJobName(createJobName(pipeline.getName(), inputNodes, reduceNode));


    return new CrunchControlledJob(
        jobID,
        job,
        new CrunchJobHooks.PrepareHook(job),
        new CrunchJobHooks.CompletionHook(job, outputPath, outputHandler.getMultiPaths(), group == null));
  }

View Full Code Here

      }
      job.setInputFormatClass(CrunchInputFormat.class);
    }
    JobNameBuilder jobNameBuilder = createJobNameBuilder(conf, pipeline.getName(), inputNodes, reduceNode, numOfJobs);


    return new CrunchControlledJob(
        jobID,
        job,
        jobNameBuilder,
        new CrunchJobHooks.PrepareHook(job),
        new CrunchJobHooks.CompletionHook(job, outputPath, outputHandler.getMultiPaths(), group == null));

View Full Code Here

      }
      job.setInputFormatClass(CrunchInputFormat.class);
    }
    job.setJobName(createJobName(pipeline.getName(), inputNodes, reduceNode));


    return new CrunchControlledJob(
        jobID,
        job,
        new CrunchJobHooks.PrepareHook(job),
        new CrunchJobHooks.CompletionHook(job, outputPath, outputHandler.getMultiPaths(), group == null));
  }

View Full Code Here

      }
      job.setInputFormatClass(CrunchInputFormat.class);
    }
    JobNameBuilder jobNameBuilder = createJobNameBuilder(conf, pipeline.getName(), inputNodes, reduceNode, numOfJobs);


    return new CrunchControlledJob(
        jobID,
        job,
        jobNameBuilder,
        allTargets,
        new CrunchJobHooks.PrepareHook(job),

View Full Code Here

      }
      job.setInputFormatClass(CrunchInputFormat.class);
    }
    job.setJobName(createJobName(conf, pipeline.getName(), inputNodes, reduceNode, numOfJobs));


    return new CrunchControlledJob(
        jobID,
        job,
        new CrunchJobHooks.PrepareHook(job),
        new CrunchJobHooks.CompletionHook(job, outputPath, outputHandler.getMultiPaths(), group == null));
  }

View Full Code Here

TOP

Related Classes of org.apache.crunch.hadoop.mapreduce.lib.jobcontrol.CrunchControlledJob

org.apache.crunch.hadoop.mapreduce.lib.jobcontrol.CrunchControlledJob.State

org.apache.crunch.impl.mr.plan.JobPrototype

org.apache.hadoop.conf.Configuration

org.apache.hadoop.fs.FileSystem

org.apache.hadoop.mapreduce.Job

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.