Examples of POSplit

org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POSplit
The MapReduce Split operator.
The assumption here is that the logical to physical translation will create this dummy operator with just the filename using which the input branch will be stored and used for loading Also the translation should make sure that appropriate filter operators are configured as outputs of this operator using the conditions specified in the LOSplit. So LOSplit will be converted into: | | | Filter1 Filter2 ... Filter3 | | ... | | | ... | ---- POSplit -... ---- This is different than the existing implementation where the POSplit writes to sidefiles after filtering and then loads the appropriate file.
The approach followed here is as good as the old approach if not better in many cases because of the availability of attachinInputs. An optimization that can ensue is if there are multiple loads that load the same file, they can be merged into one and then the operators that take input from the load can be stored. This can be used when the mapPlan executes to read the file only once and attach the resulting tuple as inputs to all the operators that take input from this load. In some cases where the conditions are exclusive and some outputs are ignored, this approach can be worse. But this leads to easier management of the Split and also allows to reuse this data stored from the split job whenever necessary.
org.apache.pig.backend.local.executionengine.physicalLayer.relationalOperators.POSplit
org.apache.pig.pen.physicalOperators.POSplit

Examples of org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POSplit

            checkPhysicalPlan(mo1.reducePlan, 1, 1, 2);
            PhysicalOperator leaf = mo1.reducePlan.getLeaves().get(0);
            
            Assert.assertTrue(leaf instanceof POSplit);


            POSplit split = (POSplit)leaf;
            
            int i = 0;
            for (PhysicalPlan p: split.getPlans()) {
                checkPhysicalPlan(p, 1, 1, 1);
                ++i;
            }


            Assert.assertEquals(i,2);

View Full Code Here

Examples of org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POSplit

            checkPhysicalPlan(mo.mapPlan, 1, 1, 4);
            PhysicalOperator leaf = mo.mapPlan.getLeaves().get(0);
            
            Assert.assertTrue(leaf instanceof POSplit);
            
            POSplit split = (POSplit)leaf;


            int i = 0;
            for (PhysicalPlan p: split.getPlans()) {
                checkPhysicalPlan(p, 1, 1, 1);
                ++i;
            }


            Assert.assertEquals(i,3);

View Full Code Here

Examples of org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POSplit

            checkPhysicalPlan(mo1.reducePlan, 1, 1, 2);
            PhysicalOperator leaf = mo1.reducePlan.getLeaves().get(0);
            
            Assert.assertTrue(leaf instanceof POSplit);


            POSplit split = (POSplit)leaf;
            
            int i = 0;
            for (PhysicalPlan p: split.getPlans()) {
                checkPhysicalPlan(p, 1, 1, 1);
                ++i;
            }


            Assert.assertEquals(i,2);

View Full Code Here

Examples of org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POSplit

        int numMerges = 0;
        
        PhysicalPlan splitterPl = isMapOnly(mr) ? mr.mapPlan : mr.reducePlan;                            
        POStore storeOp = (POStore)splitterPl.getLeaves().get(0); 
        
        POSplit splitOp = null;
        
        // case 3: multiple splittees and at least one of them is map-only
        if (mappers.size() > 0) {                
            splitOp = getSplit();  
            int n = mergeAllMapOnlySplittees(mappers, mr, splitOp);            
            
            log.info("Merged " + n + " map-only splittees.");
            
            numMerges += n;   
        }            
        
        if (mapReducers.size() > 0) {
            
            boolean isMapOnly = isMapOnly(mr);
            int merged = 0;
            
            // case 4: multiple splittees and at least one of them has reducer  
            //         and the splitter is map-only   
            if (isMapOnly) {
                         
                PhysicalOperator leaf = splitterPl.getLeaves().get(0);
                                                            
                splitOp = (leaf instanceof POStore) ? getSplit() : (POSplit)leaf;
                    
                merged = mergeMapReduceSplittees(mapReducers, mr, splitOp);  
            }
            
            // case 5: multiple splittees and at least one of them has reducer
            //         and splitter has reducer
            else {
                
                merged = mergeMapReduceSplittees(mapReducers, mr);  
                
            }
            
            log.info("Merged " + merged + " map-reduce splittees.");
            
            numMerges += merged;      
        }
        
        // Finally, add original store to the split operator 
        // if there is splittee that hasn't been merged into the splitter
        if (splitOp != null 
                && (numMerges < numSplittees)) {


            PhysicalPlan storePlan = new PhysicalPlan();
            try {
                storePlan.addAsLeaf(storeOp);
                splitOp.addPlan(storePlan);
            } catch (PlanException e) {
                int errCode = 2129;
                String msg = "Internal Error. Unable to add store to the split plan for optimization.";
                throw new OptimizerException(msg, errCode, PigException.BUG, e);
            }

View Full Code Here

Examples of org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POSplit

                // multiple splittees have different map key types
                if (!sameKeyType) {
                    lr.setKeyType(DataType.TUPLE);
                }
            } else if (leaf instanceof POSplit) {
                POSplit spl = (POSplit)leaf;
                index = setIndexOnLRInSplit(index, spl, sameKeyType);
            }
        }


        return index;

View Full Code Here

Examples of org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POSplit

            // merged earlier), then we want the POLocalRearrange
            // operators in the split to have indices 1, 2 ...
            // essentially we are flattening the index numbers
            // across all POLocalRearranges in all merged map plans
            // including nested ones in POSplit
            POSplit spl = (POSplit)leaf;
            curIndex = setIndexOnLRInSplit(index, spl, sameKeyType);
        }
                    
        splitOp.addPlan(pl);

View Full Code Here

Examples of org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POSplit

    private boolean isSingleLoadMapperPlan(PhysicalPlan pl) {
        return (pl.getRoots().size() == 1);
    }
    
    private POSplit getSplit(){
        return new POSplit(new OperatorKey(scope, nig.getNextNodeId(scope)));
    }

View Full Code Here

Examples of org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POSplit

            checkPhysicalPlan(mo1.reducePlan, 1, 1, 2);
            PhysicalOperator leaf = mo1.reducePlan.getLeaves().get(0);
            
            Assert.assertTrue(leaf instanceof POSplit);


            POSplit split = (POSplit)leaf;
            
            int i = 0;
            for (PhysicalPlan p: split.getPlans()) {
                checkPhysicalPlan(p, 1, 1, 1);
                ++i;
            }


            Assert.assertEquals(i,2);

View Full Code Here

Examples of org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POSplit

            checkPhysicalPlan(mo.mapPlan, 1, 1, 4);
            PhysicalOperator leaf = mo.mapPlan.getLeaves().get(0);
            
            Assert.assertTrue(leaf instanceof POSplit);
            
            POSplit split = (POSplit)leaf;


            int i = 0;
            for (PhysicalPlan p: split.getPlans()) {
                checkPhysicalPlan(p, 1, 1, 1);
                ++i;
            }


            Assert.assertEquals(i,3);

View Full Code Here

Examples of org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POSplit


        int numMerges = 0;


        PhysicalPlan splitterPl = isMapOnly(mr) ? mr.mapPlan : mr.reducePlan;
        POStore storeOp = (POStore)splitterPl.getLeaves().get(0);
        POSplit splitOp = null;


        // case 3: multiple splittees and at least one of them is map-only
        if (mappers.size() > 0) {
            splitOp = getSplit();
            int n = mergeAllMapOnlySplittees(mappers, mr, splitOp);
            log.info("Merged " + n + " map-only splittees.");
            numMerges += n;
        }


        if (mapReducers.size() > 0) {
            boolean isMapOnly = isMapOnly(mr);
            int merged = 0;


            // case 4: multiple splittees and at least one of them has reducer
            //         and the splitter is map-only
            if (isMapOnly) {
                PhysicalOperator leaf = splitterPl.getLeaves().get(0);
                splitOp = (leaf instanceof POStore) ? getSplit() : (POSplit)leaf;
                merged = mergeMapReduceSplittees(mapReducers, mr, splitOp);
            }


            // case 5: multiple splittees and at least one of them has reducer
            //         and splitter has reducer
            else {
                merged = mergeMapReduceSplittees(mapReducers, mr);
            }


            log.info("Merged " + merged + " map-reduce splittees.");
            numMerges += merged;
        }


        // Finally, add original store to the split operator
        // if there is splittee that hasn't been merged into the splitter
        if (splitOp != null
                && (numMerges < numSplittees)) {


            PhysicalPlan storePlan = new PhysicalPlan();
            try {
                storePlan.addAsLeaf(storeOp);
                splitOp.addPlan(storePlan);
            } catch (PlanException e) {
                int errCode = 2129;
                String msg = "Internal Error. Unable to add store to the split plan for optimization.";
                throw new OptimizerException(msg, errCode, PigException.BUG, e);
            }

View Full Code Here

0 1 2 3 4 5

TOP

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.