Examples of gov.nysenate.openleg.processors.DataProcessor

gov.nysenate.openleg.processors.DataProcessor

Contains an implementation for each step of the Open Legislation data processing pipeline.

{@link #stage(File,File) Stage}: Grab a distinct set of data files files to operate on. We are constantly receiving SOBI files so it is helpful to isolate the ones we are currently processing
{@link #collate(File) Collate}: Each SOBI file can be thought of as a collection of sub documents. The collate step extracts these files, converts them to UTF-8 encoding, and writes them to file. Writing to file is useful for debugging because it provides a record of what data was sent into each processor.
{@link #ingest(File,Storage) Ingest}: Process collated files in chronological order based on the time stamp encoded into the SOBI file name. Delegates file processing based on document type.
{@link #push(Storage,List,List) Push}: Pushes a set of changes out to a list of services.
{@link #archive(File,File) Archive}: Archives all working files for reference when going through logs in the future.

@author GraylinKim

                System.exit(1);
            }
        }


        Environment env = Application.getEnvironment();
        DataProcessor process = new DataProcessor();
        for (String task : tasks) {
            // TODO: With Java7 we can make this a switch case
            if (task.equals("stage")) {
                process.stage(env.getStagingDirectory(), env.getWorkingDirectory());
            }
            else if (task.equals("collate")) {
                process.collate(env.getWorkingDirectory());
            }
            else if (task.equals("ingest")) {
                process.ingest(env.getWorkingDirectory(), Application.getStorage());
                if (changeFile != null) {
                    ChangeLogger.writeToFile(changeFile);
                }
            }
            else if (task.equals("push")) {
                if (ChangeLogger.getChangeLog().isEmpty()) {
                    if (changeFile != null) {
                        ChangeLogger.readFromFile(changeFile);
                    }
                    else {
                        System.err.println("Unable to push with an empty change log.");
                    }
                }
                process.push(Application.getStorage(), ChangeLogger.getEntries(), services);
            }
            else if (task.equalsIgnoreCase("archive")) {
                process.archive(env.getWorkingDirectory(), env.getArchiveDirectory());
            }
            else {
                System.err.println("Invalid task.");
                System.exit(1);
            }

        // push changes to lucene and varnish
        ArrayList<ServiceBase> services = new ArrayList<ServiceBase>();
        services.add(new Lucene());
        services.add(new Varnish("127.0.0.1", 80));


        DataProcessor process = new DataProcessor();
        process.push(storage, ChangeLogger.getEntries(), services);


        // Add the bill to the unpublished bill list
        UnpublishListManager unpublishListManager = new UnpublishListManager();
        unpublishListManager.addUnpublishedBill(billId + '-' + sessionYear);
    }

Examples of gov.nysenate.openleg.processors.DataProcessor

Related Classes of gov.nysenate.openleg.processors.DataProcessor