Examples of org.apache.hadoop.io.compress.bzip2.CBZip2InputStream

Package org.apache.hadoop.io.compress.bzip2

Examples of org.apache.hadoop.io.compress.bzip2.CBZip2InputStream

org.apache.tools.bzip2.CBZip2InputStream
An input stream that decompresses from the BZip2 format (without the file header chars) to be read as any other stream.
The decompression requires large amounts of memory. Thus you should call the {@link #close() close()} method as soon as possible, to forceCBZip2InputStream to release the allocated memory. See {@link CBZip2OutputStream CBZip2OutputStream} for information about memoryusage.

CBZip2InputStream reads bytes from the compressed source stream via the single byte {@link java.io.InputStream#read() read()} method exclusively.Thus you should consider to use a buffered source stream.

This Ant code was enhanced so that it can de-compress blocks of bzip2 data. Current position in the stream is an important statistic for Hadoop. For example in LineRecordReader, we solely depend on the current position in the stream to know about the progess. The notion of position becomes complicated for compressed files. The Hadoop splitting is done in terms of compressed file. But a compressed file deflates to a large amount of data. So we have handled this problem in the following way. On object creation time, we find the next block start delimiter. Once such a marker is found, the stream stops there (we discard any read compressed data in this process) and the position is updated (i.e. the caller of this class will find out the stream location). At this point we are ready for actual reading (i.e. decompression) of data. The subsequent read calls give out data. The position is updated when the caller of this class has read off the current block + 1 bytes. In between the block reading, position is not updated. (We can only update the postion on block boundaries).

Instances of this class are not threadsafe.

        if (source.lastModified() > dest.lastModified()) {
            log("Expanding " + source.getAbsolutePath() + " to "
                + dest.getAbsolutePath());


            FileOutputStream out = null;
            CBZip2InputStream zIn = null;
            InputStream fis = null;
            BufferedInputStream bis = null;
            try {
                out = new FileOutputStream(dest);
                fis = srcResource.getInputStream();
                bis = new BufferedInputStream(fis);
                int b = bis.read();
                if (b != 'B') {
                    throw new BuildException("Invalid bz2 file.", getLocation());
                }
                b = bis.read();
                if (b != 'Z') {
                    throw new BuildException("Invalid bz2 file.", getLocation());
                }
                zIn = new CBZip2InputStream(bis);
                byte[] buffer = new byte[8 * 1024];
                int count = 0;
                do {
                    out.write(buffer, 0, count);
                    count = zIn.read(buffer, 0, buffer.length);
                } while (count != -1);
            } catch (IOException ioe) {
                String msg = "Problem expanding bzip2 " + ioe.getMessage();
                throw new BuildException(msg, ioe, getLocation());
            } finally {

View Full Code Here

    int cB = reader.readInt8();
    int cZ = reader.readInt8();
    if (cB != 'B' || cZ != 'Z') {
      throw new IOException("No BZip2 stream found");
    }
    is = new CBZip2InputStream(new InputStream() {


      int remaining = compressedLength - 2;
      
      @Override
      public int read() throws IOException {

View Full Code Here

            public InputStream getStream() throws IOException {
                InputStream raw = src.getStream();
                // A BZip2 file starts with "BZ", but CBZip2InputStream
                // expects the magic bytes to be stripped off first.
                if (raw.read() == 'B' && raw.read() == 'Z') {
                    return new BufferedInputStream(new CBZip2InputStream(raw));
                } else {
                    throw new IOException("Not BZIP2 format");
                }
            }
        };

View Full Code Here

        InputStream actualIn =
            new BufferedInputStream(new FileInputStream(actualFile));
        assertEquals((byte) 'B', actualIn.read());
        assertEquals((byte) 'Z', actualIn.read());


        originalIn = new CBZip2InputStream(originalIn);
        actualIn   = new CBZip2InputStream(actualIn);


        while (true) {
            int expected = originalIn.read();
            int actual   = actualIn.read();
            if (expected >= 0) {

View Full Code Here

                        if (istream.read() != magic[i]) {
                            throw new BuildException(
                                                     "Invalid bz2 file." + name);
                        }
                    }
                    return new CBZip2InputStream(istream);
                }
            }
            return istream;
        }

View Full Code Here

        for (int i = 0; i < MAGIC.length; i++) {
            if (in.read() != MAGIC[i]) {
                throw new IOException("Invalid bz2 stream.");
            }
        }
        return new CBZip2InputStream(in);
    }

View Full Code Here

        if (source.lastModified() > dest.lastModified()) {
            log("Expanding " + source.getAbsolutePath() + " to "
                + dest.getAbsolutePath());


            FileOutputStream out = null;
            CBZip2InputStream zIn = null;
            InputStream fis = null;
            BufferedInputStream bis = null;
            try {
                out = new FileOutputStream(dest);
                fis = srcResource.getInputStream();
                bis = new BufferedInputStream(fis);
                int b = bis.read();
                if (b != 'B') {
                    throw new BuildException("Invalid bz2 file.", getLocation());
                }
                b = bis.read();
                if (b != 'Z') {
                    throw new BuildException("Invalid bz2 file.", getLocation());
                }
                zIn = new CBZip2InputStream(bis);
                byte[] buffer = new byte[BUFFER_SIZE];
                int count = 0;
                do {
                    out.write(buffer, 0, count);
                    count = zIn.read(buffer, 0, buffer.length);
                } while (count != -1);
            } catch (IOException ioe) {
                String msg = "Problem expanding bzip2 " + ioe.getMessage();
                throw new BuildException(msg, ioe, getLocation());
            } finally {

View Full Code Here

            if (this.sourcefile.getName().endsWith(".bz2")) {
                int b = is.read();
                if (b != 'B') throw new IOException("Invalid bz2 content.");
                b = is.read();
                if (b != 'Z') throw new IOException("Invalid bz2 content.");
                is = new CBZip2InputStream(is);
            } else if (this.sourcefile.getName().endsWith(".gz")) {
                is = new GZIPInputStream(is);
            }
            final BufferedReader r = new BufferedReader(new java.io.InputStreamReader(is, "UTF-8"), 4 * 1024 * 1024);
            String t;

View Full Code Here

        if (compression == null) {
          // Do nothing
        } else
        if ("bzip2".equals(compression)) {
          skipFully(input, 2);
          input = new CBZip2InputStream(input);
        } else
        if ("gzip".equals(compression)) {
          input = new GZIPInputStream(input);
        }
        lastInputStream = new XarInputStream(entry, input);

View Full Code Here

                        if (istream.read() != magic[i]) {
                            throw new BuildException(
                                "Invalid bz2 file." + file.toString());
                        }
                    }
                    return new CBZip2InputStream(istream);
                }
            }
            return istream;
        }

View Full Code Here

0 1 2 3 4 5 6 7 8 9

TOP

Related Classes of org.apache.hadoop.io.compress.bzip2.CBZip2InputStream

at.molindo.utils.io.StreamUtils

cc.twittertools.index.IndexStatuses

com.endgame.binarypig.util.BuildSequenceFileFromArchive

com.google.gwt.ant.taskdefs.TarCat$UntarCompressionMethod

com.google.refine.importing.ImportingUtilities

com.packetloop.packetpig.loaders.pcap.PcapRecordReader

com.packetloop.packetpig.loaders.pcap.StreamingPcapRecordReader

edu.umd.cloud9.collection.wikipedia.WikipediaPagesBz2InputStream

er.woinstaller.archiver.XarFile

mwt.wow.mpq.BZip2Reader

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.