Examples of cascading.scheme.hadoop.WritableSequenceFile

cascading.scheme.hadoop.WritableSequenceFile
Class WritableSequenceFile is a sub-class of {@link SequenceFile} that reads and writes values of the given{@code writableType} {@code Class}, instead of {@link Tuple} instances used by default in SequenceFile.
This Class is a convenience for those who need to read/write specific types from existing sequence files without them being wrapped in a Tuple instance.
Note due to the nature of sequence files, only one type can be stored in the key and value positions, they they can be uniquely different types (LongWritable, Text).
If keyType is null, valueType must not be null, and vice versa, assuming you only wish to store a single value.
{@link NullWritable} is used as the empty type for either a null keyType or valueType.

          Tap writableSeqFileSink = null;
          Pipe writableSeqFileDataPipe = null;
            if (!options.isLocalPlatformMode()) {
                writableSeqFileDataPipe = new Pipe("writable seqfile data", new Each(parsePipe.getTailPipe(), new CreateWritableSeqFileData()));
                BasePath writableSeqFileDataPath = platform.makePath(curWorkingDirPath, CrawlConfig.EXTRACTED_TEXT_SUBDIR_NAME);
                WritableSequenceFile writableSeqScheme = new WritableSequenceFile(new Fields(CrawlConfig.WRITABLE_SEQ_FILE_KEY_FN, CrawlConfig.WRITABLE_SEQ_FILE_VALUE_FN), Text.class, Text.class);
                writableSeqFileSink = platform.makeTap(writableSeqScheme, writableSeqFileDataPath, SinkMode.REPLACE);
            }
        
        Pipe urlFromOutlinksPipe = new Pipe("url from outlinks", parsePipe.getTailPipe());
        urlFromOutlinksPipe = new Each(urlFromOutlinksPipe, new CreateUrlDatumFromOutlinksFunction(new SimpleUrlNormalizer(), new SimpleUrlValidator()));

View Full Code Here

                //if(s.getPath().toString().contains("part-")){//found a part-xxxxx file
                if(s.getPath().getName().matches("^part.*")){//found a part-xxxxx file
                    Path filePath = new Path(s.getPath().toString());
                    //Tap t = new Hfs( inFields, filePath.toString());
                    //Tap t = new Hfs(new TextLine(), filePath.toString(), true);
                    Tap t = new Hfs( new WritableSequenceFile( f, LongWritable.class, VectorWritable.class ), filePath.toString() );
                    if( s.getLen() != 0 ){// then part file is not empty
                        all.add(t);
                    }
                }
            }

View Full Code Here

TOP

Related Classes of cascading.scheme.hadoop.WritableSequenceFile

bixo.examples.crawl.DemoCrawlWorkflow

cascading.tuple.TupleEntry

finderbots.recommenders.hadoop.WriteDRMsToSolr

org.apache.hadoop.io.Writable

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.