Examples of addOutputKey()

Examples of org.apache.pig.backend.hadoop.executionengine.tez.plan.operator.POValueOutputTez.addOutputKey()

286
287
288
289
290
291
292
293
294
295
296
                    if (storeSeen.containsKey(store)) {
                        storeSeen.get(store).addOutputKey(tezOp.getOperatorKey().toString());
                    } else {
                        POValueOutputTez output = new POValueOutputTez(OperatorKey.genOpKey(scope));
                        output.addOutputKey(tezOp.getOperatorKey().toString());
                        from.plan.remove(from.plan.getOperator(store.getOperatorKey()));
                        from.plan.addAsLeaf(output);
                        storeSeen.put(store, output);

                        //Remove unused store filename
View Full Code Here

Examples of org.apache.pig.backend.hadoop.executionengine.tez.plan.operator.POValueOutputTez.addOutputKey()

836
837
838
839
840
841
842
843
844
845
846
            // Explicitly set the parallelism for the new vertex to 1.
            curTezOp.setRequestedParallelism(1);
            curTezOp.setDontEstimateParallelism(true);

            output.addOutputKey(curTezOp.getOperatorKey().toString());
            // LIMIT does not make any ordering guarantees and this is unsorted shuffle.
            TezEdgeDescriptor edge = curTezOp.inEdges.get(prevOp.getOperatorKey());
            TezCompilerUtil.configureValueOnlyTupleOutput(edge, DataMovementType.SCATTER_GATHER);

            // Limit after order by with scalar expression
View Full Code Here

Examples of org.apache.pig.backend.hadoop.executionengine.tez.plan.operator.POValueOutputTez.addOutputKey()

1676
1677
1678
1679
1680
1681
1682
1683
1684
1685
1686
                joinJobs[i].setSampleOperator(sampleJobPair.first);

                // Configure broadcast edges for distribution map
                edge = TezCompilerUtil.connect(tezPlan, sampleJobPair.first, joinJobs[i]);
                TezCompilerUtil.configureValueOnlyTupleOutput(edge, DataMovementType.BROADCAST);
                sampleOut.addOutputKey(joinJobs[i].getOperatorKey().toString());

                // Configure skewed partitioner for join
                edge = joinJobs[2].inEdges.get(joinJobs[i].getOperatorKey());
                edge.partitionerClass = SkewedPartitionerTez.class;
            }
View Full Code Here

Examples of org.apache.pig.backend.hadoop.executionengine.tez.plan.operator.POValueOutputTez.addOutputKey()

2270
2271
2272
2273
2274
2275
2276
2277
2278
2279
2280
            lrSample.setOutputKey(quantJobParallelismPair.first.getOperatorKey().toString());

            edge = TezCompilerUtil.connect(tezPlan, quantJobParallelismPair.first, sortOpers[0]);
            TezCompilerUtil.configureValueOnlyTupleOutput(edge, DataMovementType.BROADCAST);
            POValueOutputTez sampleOut = (POValueOutputTez)quantJobParallelismPair.first.plan.getLeaves().get(0);
            sampleOut.addOutputKey(sortOpers[0].getOperatorKey().toString());
            sortOpers[0].setSampleOperator(quantJobParallelismPair.first);

            edge = TezCompilerUtil.connect(tezPlan, sortOpers[0], sortOpers[1]);
            edge.partitionerClass = WeightedRangePartitionerTez.class;
View Full Code Here

Examples of org.apache.pig.backend.hadoop.executionengine.tez.plan.operator.POValueOutputTez.addOutputKey()

2305
2306
2307
2308
2309
2310
2311
2312
2313
2314
2315
                limitOper.markLimitAfterSort();

                edge = TezCompilerUtil.connect(tezPlan, sortOpers[1], limitOper);
                // LIMIT in this case should be ordered. So we output unordered with key as task index
                // and on the input we use OrderedGroupedKVInput to do ordered merge to retain sorted order.
                output.addOutputKey(limitOper.getOperatorKey().toString());
                output.setTaskIndexWithRecordIndexAsKey(true);
                edge = curTezOp.inEdges.get(sortOpers[1].getOperatorKey());
                TezCompilerUtil.configureValueOnlyTupleOutput(edge, DataMovementType.SCATTER_GATHER);
                // POValueOutputTez will write key (task index, record index) in
                // sorted order. So using UnorderedKVOutput instead of OrderedPartitionedKVOutput.
View Full Code Here

Examples of org.apache.pig.backend.hadoop.executionengine.tez.plan.operator.POValueOutputTez.addOutputKey()

2355
2356
2357
2358
2359
2360
2361
2362
2363
2364
2365
                splitOp.plan.addAsLeaf(output);
            }
            curTezOp = getTezOp();
            curTezOp.setSplitParent(splitOp.getOperatorKey());
            tezPlan.add(curTezOp);
            output.addOutputKey(curTezOp.getOperatorKey().toString());
            TezEdgeDescriptor edge = TezCompilerUtil.connect(tezPlan, splitOp, curTezOp);
            //TODO shared edge once support is available in Tez
            TezCompilerUtil.configureValueOnlyTupleOutput(edge, DataMovementType.ONE_TO_ONE);
            curTezOp.setRequestedParallelismByReference(splitOp);
            POValueInputTez input = new POValueInputTez(OperatorKey.genOpKey(scope));
View Full Code Here
TOP
Copyright © 2018 www.massapi.com. All rights reserved.
All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.