Examples of uk.ac.ucl.panda.indexing.io.DocumentsWriter

• uk.ac.ucl.panda.indexing.io.DocumentsWriter
  This class accepts multiple added documents and directly writes a single segment file. It does this more efficiently than creating a single segment per document (with DocumentWriter) and doing standard merges on those segments. When a document is added, its stored fields (if any) and term vectors (if any) are immediately written to the Directory (ie these do not consume RAM). The freq/prox postings are accumulated into a Postings hash table keyed by term. Each entry in this hash table holds a separate byte stream (allocated as incrementally growing slices into large shared byte[] arrays) for freq and prox, that contains the postings data for multiple documents. If vectors are enabled, each unique term for each document also allocates a PostingVector instance to similarly track the offsets & positions byte stream. Once the Postings hash is full (ie is consuming the allowed RAM) or the number of added docs is large enough (in the case we are flushing by doc count instead of RAM usage), we create a real segment and flush it to disk and reset the Postings hash. In adding a document we first organize all of its fields by field name. We then process field by field, and record the Posting hash per-field. After each field we flush its term vectors. When it's time to flush the full segment we first sort the fields by name, and then go field by field and sorts its postings. Threads: Multiple threads are allowed into addDocument at once. There is an initial synchronized call to getThreadState which allocates a ThreadState for this thread. The same thread will get the same ThreadState over time (thread affinity) so that if there are consistent patterns (for example each thread is indexing a different content source) then we make better use of RAM. Then processDocument is called on that ThreadState without synchronization (most of the "heavy lifting" is in this call). Finally the synchronized "finishDocument" is called to flush changes to the directory. Each ThreadState instance has its own Posting hash. Once we're using too much RAM, we flush all Posting hashes to a segment by merging the docIDs in the posting lists for the same term across multiple thread states (see writeSegment and appendPostings). When flush is called by IndexWriter, or, we flush internally when autoCommit=false, we forcefully idle all threads and flush only once they are all idle. This means you can call flush with a given thread even while other threads are actively adding/deleting documents. Exceptions: Because this class directly updates in-memory posting lists, and flushes stored fields and term vectors directly to files in the directory, there are certain limited times when an exception can corrupt this state. For example, a disk full while flushing stored fields leaves this file in a corrupt state. Or, an OOM exception while appending to the in-memory posting lists can corrupt that posting list. We call such exceptions "aborting exceptions". In these cases we must call abort() to discard all docs added since the last flush. All other exceptions ("non-aborting exceptions") can still partially update the index structures. These updates are consistent, but, they represent only a part of the document seen up until the exception was hit. When this happens, we immediately mark the document as deleted so that the document is always atomically ("all or none") added to the index.

Related Classes of uk.ac.ucl.panda.indexing.io.DocumentsWriter