List of applyPTypeTransforms() Examples

Examples of applyPTypeTransforms()

org.apache.crunch.types.Converter.applyPTypeTransforms()
If true, convert the inputs or outputs from this {@code Converter} instancebefore (for outputs) or after (for inputs) using the associated PType#getInputMapFn and PType#getOutputMapFn calls.

Examples of org.apache.crunch.types.Converter.applyPTypeTransforms()

          job.getConfiguration(),
          CrunchInputFormat.class,
          converter.getKeyClass(),
          converter.getValueClass());
      input.rdd().setName(source.toString());
      MapFn mapFn = converter.applyPTypeTransforms() ? source.getType().getInputMapFn() : IdentityFn.getInstance();
      return input
          .map(new InputConverterFunction(source.getConverter()))
          .map(new MapFunction(mapFn, runtime.getRuntimeContext()));
    } catch (IOException e) {
      throw new RuntimeException(e);

View Full Code Here

Examples of org.apache.crunch.types.Converter.applyPTypeTransforms()

    return Lists.newArrayList();
  }


  public static <K, V> DoNode createGroupingNode(String name, PGroupedTableType<K, V> ptype) {
    Converter groupingConverter = ptype.getGroupingConverter();
    DoFn<?, ?> fn = groupingConverter.applyPTypeTransforms() ? ptype.getOutputMapFn() : IdentityFn.getInstance();
    return new DoNode(fn, name, ptype, NO_CHILDREN, ptype.getGroupingConverter(), null, null);
  }


  public static DoNode createOutputNode(String name, Converter outputConverter, PType<?> ptype) {
    DoFn<?, ?> fn = outputConverter.applyPTypeTransforms() ? ptype.getOutputMapFn() : IdentityFn.getInstance();

View Full Code Here

Examples of org.apache.crunch.types.Converter.applyPTypeTransforms()

  }


  public static <S> DoNode createInputNode(Source<S> source) {
    Converter srcConverter = source.getConverter();
    PType<?> ptype = source.getType();
    DoFn<?, ?> fn = srcConverter.applyPTypeTransforms() ? ptype.getInputMapFn() : IdentityFn.getInstance();
    return new DoNode(fn, source.toString(), ptype, allowsChildren(), null, source, null);
  }


  public boolean isOutputNode() {
    return outputConverter != null;

View Full Code Here

Examples of org.apache.crunch.types.Converter.applyPTypeTransforms()

            Converter c = t.getConverter(ptype);
            IdentityFn ident = IdentityFn.getInstance();
            JavaPairRDD<?, ?> outRDD;
            if (rdd instanceof JavaRDD) {
              outRDD = ((JavaRDD) rdd)
                  .map(new MapFunction(c.applyPTypeTransforms() ? ptype.getOutputMapFn() : ident, ctxt))
                  .mapToPair(new OutputConverterFunction(c));
            } else {
              outRDD = ((JavaPairRDD) rdd)
                  .map(new PairMapFunction(c.applyPTypeTransforms() ? ptype.getOutputMapFn() : ident, ctxt))
                  .mapToPair(new OutputConverterFunction(c));

View Full Code Here

Examples of org.apache.crunch.types.Converter.applyPTypeTransforms()

              outRDD = ((JavaRDD) rdd)
                  .map(new MapFunction(c.applyPTypeTransforms() ? ptype.getOutputMapFn() : ident, ctxt))
                  .mapToPair(new OutputConverterFunction(c));
            } else {
              outRDD = ((JavaPairRDD) rdd)
                  .map(new PairMapFunction(c.applyPTypeTransforms() ? ptype.getOutputMapFn() : ident, ctxt))
                  .mapToPair(new OutputConverterFunction(c));
            }
            try {
              Job job = new Job(conf);
              if (t instanceof PathTarget) {

View Full Code Here

Examples of org.apache.crunch.types.Converter.applyPTypeTransforms()

          job.getConfiguration(),
          CrunchInputFormat.class,
          converter.getKeyClass(),
          converter.getValueClass());
      input.rdd().setName(source.toString());
      MapFn mapFn = converter.applyPTypeTransforms() ? source.getType().getInputMapFn() : IdentityFn.getInstance();
      return input
          .map(new InputConverterFunction(source.getConverter()))
          .map(new PairMapFunction(mapFn, runtime.getRuntimeContext()));
    } catch (IOException e) {
      throw new RuntimeException(e);

View Full Code Here

TOP

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.