Examples of OutputCollector

backtype.storm.task.OutputCollector
This output collector exposes the API for emitting tuples from an IRichBolt. This is the core API for emitting tuples. For a simpler API, and a more restricted form of stream processing, see IBasicBolt and BasicOutputCollector.
com.btaz.datautil.files.mapreduce.OutputCollector
User: msundell
com.btaz.util.mr.OutputCollector
User: msundell
org.apache.hadoop.mapred.OutputCollector
Collects the <key, value> pairs output by {@link Mapper}s and {@link Reducer}s.
OutputCollector is the generalization of the facility provided by the Map-Reduce framework to collect data output by either the Mapper or the Reducer i.e. intermediate outputs or the output of the job.

Examples of org.apache.hadoop.mapred.OutputCollector


  public void reduce(final WritableComparable key, Iterator values,
    final OutputCollector output, Reporter reporter)
    throws IOException
  {
    OutputCollector oc = new OutputCollector()
    {
      public void collect(WritableComparable k, Writable v)
        throws IOException
      {
        // Substitute original key in place of passed k.  The original

View Full Code Here

Examples of org.apache.hadoop.mapred.OutputCollector

    final OutputCollector output, Reporter r)
    throws IOException
  {
    final String collection = Nutchwax.getCollectionFromWaxKey(key);
    
    final OutputCollector oo = new OutputCollector()
    {
      public void collect(WritableComparable k, Writable v)
        throws IOException
      {
        output.collect(Nutchwax.generateWaxKey(k, collection), v);

View Full Code Here

Examples of org.apache.hadoop.mapred.OutputCollector

        // called at all in this task). If reducer still generates output,
        // which is very uncommon and we may not have to support this case.
        // So we don't write this output to HDFS, but we consume/collect
        // this output just to avoid reducer hanging forever.


        OutputCollector collector = new OutputCollector() {
          public void collect(Object key, Object value)
            throws IOException {
            //just consume it, no need to write the record anywhere
          }
        };

View Full Code Here

Examples of org.apache.hadoop.mapred.OutputCollector

        // create a new record writer
        writer = outputFormat.getRecordWriter(FileSystem.get(outputConf), 
                                              outputConf, fileName, reporter);


        // return an output collector using the writer we just created.
        return new StoreFuncAdaptor(new OutputCollector() 
            {
                public void collect(Object key, Object value) throws IOException {
                    writer.write(key,value);
                }
            });

View Full Code Here

Examples of org.apache.hadoop.mapred.OutputCollector

   * Handle the end of the input by closing down the application.
   */
  public void close() throws IOException {
    // if we haven't started the application, we have nothing to do
    if (isOk) {
      OutputCollector nullCollector = new OutputCollector() {
        public void collect(WritableComparable key, 
                            Writable value) throws IOException {
          // NULL
        }
      };

View Full Code Here

Examples of org.apache.hadoop.mapred.OutputCollector

  private void runOldCombiner(final TezRawKeyValueIterator rawIter, final Writer writer) throws IOException {
    Class<? extends Reducer> reducerClazz = (Class<? extends Reducer>) conf.getClass("mapred.combiner.class", null, Reducer.class);
    
    Reducer combiner = ReflectionUtils.newInstance(reducerClazz, conf);
    
    OutputCollector collector = new OutputCollector() {
      @Override
      public void collect(Object key, Object value) throws IOException {
        writer.append(key, value);
      }
    };

View Full Code Here

Examples of org.apache.hadoop.mapred.OutputCollector

    
    updateJobWithSplit(job, inputSplit);


    RecordReader in = new OldRecordReader(input);


    OutputCollector collector = new OldOutputCollector(output);


    MapRunnable runner =
        (MapRunnable)ReflectionUtils.newInstance(job.getMapRunnerClass(), job);


    runner.run(in, collector, (Reporter)reporter);

View Full Code Here

Examples of org.apache.hadoop.mapred.OutputCollector

    Reducer reducer =
        ReflectionUtils.newInstance(job.getReducerClass(), job);


    // make output collector


    OutputCollector collector =
        new OutputCollector() {
      public void collect(Object key, Object value)
          throws IOException {
        output.write(key, value);
      }
    };

View Full Code Here

Examples of org.apache.hadoop.mapred.OutputCollector

        // called at all in this task). If reducer still generates output,
        // which is very uncommon and we may not have to support this case.
        // So we don't write this output to HDFS, but we consume/collect
        // this output just to avoid reducer hanging forever.


        OutputCollector collector = new OutputCollector() {
          public void collect(Object key, Object value)
            throws IOException {
            //just consume it, no need to write the record anywhere
          }
        };

View Full Code Here

Examples of org.apache.hadoop.mapred.OutputCollector

      // validate input split
      InputSplit split = splits[i];
      Assert.assertTrue(split instanceof TableSnapshotInputFormat.TableSnapshotRegionSplit);


      // validate record reader
      OutputCollector collector = mock(OutputCollector.class);
      Reporter reporter = mock(Reporter.class);
      RecordReader<ImmutableBytesWritable, Result> rr = tsif.getRecordReader(split, job, reporter);


      // validate we can read all the data back
      ImmutableBytesWritable key = rr.createKey();

View Full Code Here

0 1 2 3 4 5

TOP

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.